2024-09-17T16:42:56.349 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T16:42:56.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T16:42:56.439 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909106 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} email: skanta@redhat.com first_in_suite: false job_id: '7909106' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvGwxjycA2AQj3hCP4Bxb/0dNIDyMbAR01O2wUQI0lw8eUc7b00QsQnccO9i8rzX+BvG8nJV1LARTcWTihnQjo= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - mon-stretch teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T16:42:56.439 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T16:42:56.439 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T16:42:56.440 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T16:42:56.441 INFO:teuthology.task.internal:Checking packages... 2024-09-17T16:42:56.461 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T16:42:56.462 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T16:42:56.462 INFO:teuthology.packaging:ref: None 2024-09-17T16:42:56.462 INFO:teuthology.packaging:tag: None 2024-09-17T16:42:56.462 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T16:42:56.462 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:42:56.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T16:42:56.604 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T16:42:56.605 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T16:42:56.613 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T16:42:56.613 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T16:42:56.666 INFO:teuthology.task.internal:Saving configuration 2024-09-17T16:42:56.676 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T16:42:56.691 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T16:42:56.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909106', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 16:38:25.899597', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvGwxjycA2AQj3hCP4Bxb/0dNIDyMbAR01O2wUQI0lw8eUc7b00QsQnccO9i8rzX+BvG8nJV1LARTcWTihnQjo='} 2024-09-17T16:42:56.714 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T16:42:56.722 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-17T16:42:56.722 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T16:42:56.776 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1596033d00>, signals=[15]) 2024-09-17T16:42:56.776 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T16:42:56.784 INFO:teuthology.task.internal:Opening connections... 2024-09-17T16:42:56.784 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-09-17T16:42:56.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T16:42:56.863 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T16:42:56.871 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-09-17T16:42:56.942 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-09-17T16:42:56.942 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-09-17T16:42:57.000 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-09-17T16:42:57.000 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-09-17T16:42:57.000 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T16:42:57.001 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T16:42:57.002 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-09-17T16:42:57.028 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T16:42:57.036 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T16:42:57.059 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T16:42:57.059 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-09-17T16:42:57.076 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T16:42:57.082 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T16:42:57.082 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-09-17T16:42:57.142 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T16:42:57.175 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T16:42:57.190 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T16:42:57.190 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T16:42:57.190 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T16:42:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.207 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-17T16:42:57.264 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-503.el9.x86_64 2024-09-17T16:42:57.265 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-503.el9.x86_64 2024-09-17T16:42:57.265 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-09-17T16:43:00.969 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.3 MB 00:01 2024-09-17T16:43:20.577 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:18 2024-09-17T16:43:26.778 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-17T16:43:28.294 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-17T16:43:29.182 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-17T16:43:34.805 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T16:43:36.795 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T16:43:36.795 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T16:43:36.892 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:43:36.903 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:43:36.903 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-17T16:43:36.903 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:43:36.903 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:43:36.903 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-09-17T16:43:36.904 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:43:36.905 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 79 M 2024-09-17T16:43:36.905 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 126 M 2024-09-17T16:43:36.905 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:43:37.139 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 302 kB/s | 36 kB 00:00 2024-09-17T16:43:44.056 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-17T16:43:49.914 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.2 MB/s | 29 MB 00:12 2024-09-17T16:43:52.964 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-17T16:43:52.964 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:43:52.964 INFO:teuthology.orchestra.run.smithi037.stdout:Total 4.9 MB/s | 79 MB 00:16 2024-09-17T16:43:53.673 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:43:54.042 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:43:54.042 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:43:55.027 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:43:55.028 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:43:57.581 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:43:58.267 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T16:43:58.348 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T16:44:01.101 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T16:44:01.235 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T16:44:06.363 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T16:44:06.427 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:11.567 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:50.740 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:50.781 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:51.203 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:51.203 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T16:44:51.203 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T16:44:51.204 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T16:44:52.074 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:44:52.075 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:44:52.448 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T16:44:52.535 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.535 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T16:44:52.535 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T16:44:52.536 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T16:44:52.536 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T16:44:52.536 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T16:44:52.536 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T16:44:52.536 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T16:44:52.594 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T16:44:52.595 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T16:44:52.595 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-09-17T16:44:52.636 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 82.el9 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8274670 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T16:44:52.637 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:44:52.638 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T16:44:52.639 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T16:44:52.639 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 5.14.0-508.el9.x86_64 2024-09-17T16:44:52.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T16:44:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-09-17T16:44:54.573 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T16:44:54.601 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-09-17T16:44:54.604 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-17T16:44:54.620 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.f6BefhiCJR 2024-09-17T16:44:54.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.f6BefhiCJR 2024-09-17T16:44:54.687 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.f6BefhiCJR 2024-09-17T16:44:54.819 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.f6BefhiCJR is 6KB 2024-09-17T16:44:54.868 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.f6BefhiCJR 2024-09-17T16:44:54.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T16:44:54.945 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T16:44:54.945 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T16:44:54.945 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T16:44:54.945 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T16:44:54.946 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T16:44:55.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-09-17T16:45:25.114 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T16:45:25.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:45:25.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T16:45:43.617 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-17T16:45:52.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:45:52.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T16:46:11.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-17T16:46:23.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:46:23.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T16:46:24.333 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-17T16:46:24.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:46:24.716 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T16:46:24.716 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-17T16:46:24.733 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-508.el9.x86_64 2024-09-17T16:46:24.733 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T16:46:24.733 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T16:46:24.733 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T16:46:25.735 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T16:46:25.735 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T16:46:25.886 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-09-17T16:46:25.912 DEBUG:teuthology.parallel:result is None 2024-09-17T16:46:25.912 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T16:46:25.958 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T16:46:25.959 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T16:46:25.974 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T16:46:25.988 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T16:46:26.021 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T16:46:26.021 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T16:46:26.050 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T16:46:26.061 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T16:46:26.061 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T16:46:26.112 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:46:26.124 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:46:26.124 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T16:46:26.136 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T16:46:26.136 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T16:46:26.188 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T16:46:26.201 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T16:46:26.201 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T16:46:26.243 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T16:46:26.322 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T16:46:26.363 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T16:46:26.442 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T16:46:26.481 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:46:26.481 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T16:46:26.642 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-09-17T16:46:26.766 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T16:46:26.914 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T16:46:26.923 INFO:teuthology.task.internal:Starting timer... 2024-09-17T16:46:26.923 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T16:46:26.949 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T16:46:26.981 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-09-17T16:46:27.014 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-09-17T16:46:27.016 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T16:46:27.016 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-09-17T16:46:27.079 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-09-17T16:46:27.080 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive'} 2024-09-17T16:46:27.080 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T16:46:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:46:27.144 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T16:46:27.144 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-09-17T16:46:27.216 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T16:46:27.230 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T16:46:27.241 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T16:46:27.242 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T16:50:17.879 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com')] 2024-09-17T16:50:17.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:50:17.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T16:50:17.958 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-17T16:50:18.037 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-17T16:50:18.038 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T16:50:18.051 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T16:50:18.051 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T16:50:18.052 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T16:50:18.109 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T16:50:18.124 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T16:50:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-09-17T16:50:18.161 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-09-17T16:50:18.176 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T16:50:18.190 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T16:50:18.251 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:50:18.256 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:50:18.256 INFO:teuthology.run_tasks:Running task install... 2024-09-17T16:50:18.265 DEBUG:teuthology.task.install:project ceph 2024-09-17T16:50:18.265 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T16:50:18.265 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T16:50:18.265 INFO:teuthology.task.install:Using flavor: default 2024-09-17T16:50:18.271 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T16:50:18.271 INFO:teuthology.task.install:extra packages: [] 2024-09-17T16:50:18.271 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T16:50:18.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:50:18.420 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T16:50:18.420 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T16:50:18.535 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T16:50:18.535 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:50:18.535 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T16:50:18.562 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T16:50:18.562 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T16:50:18.630 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T16:50:18.707 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T16:50:18.773 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-09-17T16:50:18.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-09-17T16:50:19.022 INFO:teuthology.orchestra.run.smithi037.stdout:45 files removed 2024-09-17T16:50:19.050 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-09-17T16:50:19.630 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-17T16:50:19.914 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-17T16:50:20.181 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-17T16:50:20.740 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-17T16:50:23.008 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-17T16:50:29.084 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-17T16:50:30.634 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-17T16:50:31.739 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-17T16:50:37.378 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T16:50:39.698 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:50:39.717 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:50:39.717 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:50:39.717 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:50:39.717 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:50:39.717 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T16:50:39.718 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T16:50:39.719 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout:Install 31 Packages 2024-09-17T16:50:39.720 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-09-17T16:50:39.721 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:50:39.721 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 62 M 2024-09-17T16:50:39.721 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:50:40.438 INFO:teuthology.orchestra.run.smithi037.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-17T16:50:40.513 INFO:teuthology.orchestra.run.smithi037.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 335 kB/s | 25 kB 00:00 2024-09-17T16:50:40.697 INFO:teuthology.orchestra.run.smithi037.stdout:(3/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 14 MB/s | 11 MB 00:00 2024-09-17T16:50:40.730 INFO:teuthology.orchestra.run.smithi037.stdout:(4/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 3.4 MB/s | 756 kB 00:00 2024-09-17T16:50:40.772 INFO:teuthology.orchestra.run.smithi037.stdout:(5/33): libradosstriper1-19.3.0-4975.g1889a165. 6.6 MB/s | 508 kB 00:00 2024-09-17T16:50:40.856 INFO:teuthology.orchestra.run.smithi037.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 541 kB/s | 45 kB 00:00 2024-09-17T16:50:40.956 INFO:teuthology.orchestra.run.smithi037.stdout:(7/33): python3-ceph-common-19.3.0-4975.g1889a1 1.4 MB/s | 143 kB 00:00 2024-09-17T16:50:41.039 INFO:teuthology.orchestra.run.smithi037.stdout:(8/33): librgw2-19.3.0-4975.g1889a165.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-17T16:50:41.065 INFO:teuthology.orchestra.run.smithi037.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 1.5 MB/s | 161 kB 00:00 2024-09-17T16:50:41.090 INFO:teuthology.orchestra.run.smithi037.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.2 MB/s | 321 kB 00:00 2024-09-17T16:50:41.115 INFO:teuthology.orchestra.run.smithi037.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 5.9 MB/s | 305 kB 00:00 2024-09-17T16:50:41.141 INFO:teuthology.orchestra.run.smithi037.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T16:50:41.358 INFO:teuthology.orchestra.run.smithi037.stdout:(13/33): ceph-common-19.3.0-4975.g1889a165.el9. 15 MB/s | 22 MB 00:01 2024-09-17T16:50:41.416 INFO:teuthology.orchestra.run.smithi037.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-09-17T16:50:41.458 INFO:teuthology.orchestra.run.smithi037.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T16:50:41.483 INFO:teuthology.orchestra.run.smithi037.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-09-17T16:50:41.509 INFO:teuthology.orchestra.run.smithi037.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-17T16:50:41.551 INFO:teuthology.orchestra.run.smithi037.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-17T16:50:41.576 INFO:teuthology.orchestra.run.smithi037.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-17T16:50:41.601 INFO:teuthology.orchestra.run.smithi037.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 458 kB/s | 42 kB 00:00 2024-09-17T16:50:41.626 INFO:teuthology.orchestra.run.smithi037.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-17T16:50:41.652 INFO:teuthology.orchestra.run.smithi037.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-17T16:50:41.677 INFO:teuthology.orchestra.run.smithi037.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-17T16:50:41.735 INFO:teuthology.orchestra.run.smithi037.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T16:50:41.760 INFO:teuthology.orchestra.run.smithi037.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T16:50:41.786 INFO:teuthology.orchestra.run.smithi037.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 280 kB/s | 26 kB 00:00 2024-09-17T16:50:41.811 INFO:teuthology.orchestra.run.smithi037.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T16:50:41.845 INFO:teuthology.orchestra.run.smithi037.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-17T16:50:41.903 INFO:teuthology.orchestra.run.smithi037.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-17T16:50:41.954 INFO:teuthology.orchestra.run.smithi037.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-17T16:50:42.029 INFO:teuthology.orchestra.run.smithi037.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T16:50:42.146 INFO:teuthology.orchestra.run.smithi037.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-17T16:50:42.205 INFO:teuthology.orchestra.run.smithi037.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 13 MB/s | 3.3 MB 00:00 2024-09-17T16:50:42.210 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:50:42.210 INFO:teuthology.orchestra.run.smithi037.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-17T16:50:42.329 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:50:42.370 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:50:42.370 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:50:43.075 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:50:43.076 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:50:44.648 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:50:44.728 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T16:50:44.760 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T16:50:44.785 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T16:50:44.808 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T16:50:44.831 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T16:50:45.020 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T16:50:45.072 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T16:50:45.427 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T16:50:45.536 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T16:50:45.630 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T16:50:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T16:50:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T16:50:45.817 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T16:50:45.838 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T16:50:45.882 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T16:50:45.952 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T16:50:45.982 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T16:50:46.046 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T16:50:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T16:50:46.190 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T16:50:46.223 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T16:50:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T16:50:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T16:50:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T16:50:46.383 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T16:50:46.412 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T16:50:46.747 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T16:50:46.788 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T16:50:46.897 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T16:50:46.926 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T16:50:46.962 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T16:50:47.484 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T16:50:47.515 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T16:50:47.587 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T16:50:47.668 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T16:50:47.701 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T16:50:50.094 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T16:50:50.130 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T16:50:50.724 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T16:50:50.766 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T16:50:50.863 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T16:50:50.863 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T16:50:50.863 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T16:50:50.863 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:50:50.893 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T16:51:01.051 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-09-17T16:51:01.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:02.092 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T16:51:02.119 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T16:51:02.120 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:02.120 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T16:51:02.120 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:51:02.120 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:51:02.120 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:02.121 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T16:51:02.211 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T16:51:02.211 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T16:51:03.422 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T16:51:03.423 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T16:51:03.424 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T16:51:03.425 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T16:51:03.426 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T16:51:03.426 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T16:51:03.426 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T16:51:03.426 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T16:51:03.426 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T16:51:04.455 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T16:51:04.455 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:04.455 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-09-17T16:51:04.455 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T16:51:04.456 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T16:51:04.457 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:04.458 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:04.459 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:04.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-09-17T16:51:05.397 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:05.535 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-09-17T16:51:05.536 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:05.537 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 46 M 2024-09-17T16:51:05.537 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 201 M 2024-09-17T16:51:05.537 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:06.007 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-17T16:51:06.032 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 658 kB/s | 162 kB 00:00 2024-09-17T16:51:06.057 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-17T16:51:07.674 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 24 MB/s | 46 MB 00:01 2024-09-17T16:51:07.676 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:07.676 INFO:teuthology.orchestra.run.smithi037.stdout:Total 22 MB/s | 46 MB 00:02 2024-09-17T16:51:07.705 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:07.728 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:07.728 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:08.115 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:08.115 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:08.640 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:08.726 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T16:51:08.777 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T16:51:08.801 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T16:51:13.413 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T16:51:13.493 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T16:51:15.741 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T16:51:15.741 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T16:51:15.741 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T16:51:15.741 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T16:51:16.048 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:16.049 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:16.289 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-09-17T16:51:16.902 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:17.045 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:17.048 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:17.048 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:17.048 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:17.048 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T16:51:17.049 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T16:51:17.050 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T16:51:17.051 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T16:51:17.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:17.052 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:17.052 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:17.052 INFO:teuthology.orchestra.run.smithi037.stdout:Install 37 Packages 2024-09-17T16:51:17.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:17.053 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 M 2024-09-17T16:51:17.053 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 109 M 2024-09-17T16:51:17.053 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:17.433 INFO:teuthology.orchestra.run.smithi037.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-17T16:51:17.616 INFO:teuthology.orchestra.run.smithi037.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-17T16:51:17.666 INFO:teuthology.orchestra.run.smithi037.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-09-17T16:51:17.900 INFO:teuthology.orchestra.run.smithi037.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 1.2 MB/s | 291 kB 00:00 2024-09-17T16:51:17.975 INFO:teuthology.orchestra.run.smithi037.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 8.8 MB/s | 4.8 MB 00:00 2024-09-17T16:51:18.151 INFO:teuthology.orchestra.run.smithi037.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-17T16:51:18.343 INFO:teuthology.orchestra.run.smithi037.stdout:(7/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 24 MB/s | 17 MB 00:00 2024-09-17T16:51:18.368 INFO:teuthology.orchestra.run.smithi037.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-17T16:51:18.418 INFO:teuthology.orchestra.run.smithi037.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-17T16:51:18.452 INFO:teuthology.orchestra.run.smithi037.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-17T16:51:18.485 INFO:teuthology.orchestra.run.smithi037.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-17T16:51:18.519 INFO:teuthology.orchestra.run.smithi037.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 768 kB/s | 135 kB 00:00 2024-09-17T16:51:18.611 INFO:teuthology.orchestra.run.smithi037.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 379 kB/s | 35 kB 00:00 2024-09-17T16:51:18.636 INFO:teuthology.orchestra.run.smithi037.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-17T16:51:18.670 INFO:teuthology.orchestra.run.smithi037.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 933 kB/s | 172 kB 00:00 2024-09-17T16:51:18.695 INFO:teuthology.orchestra.run.smithi037.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 919 kB/s | 77 kB 00:00 2024-09-17T16:51:18.721 INFO:teuthology.orchestra.run.smithi037.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 606 kB/s | 51 kB 00:00 2024-09-17T16:51:18.746 INFO:teuthology.orchestra.run.smithi037.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-17T16:51:18.796 INFO:teuthology.orchestra.run.smithi037.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-17T16:51:18.822 INFO:teuthology.orchestra.run.smithi037.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-17T16:51:18.855 INFO:teuthology.orchestra.run.smithi037.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2024-09-17T16:51:18.880 INFO:teuthology.orchestra.run.smithi037.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 462 kB/s | 11 kB 00:00 2024-09-17T16:51:18.905 INFO:teuthology.orchestra.run.smithi037.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-17T16:51:18.925 INFO:teuthology.orchestra.run.smithi037.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 393 kB/s | 18 kB 00:00 2024-09-17T16:51:18.951 INFO:teuthology.orchestra.run.smithi037.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 514 kB/s | 23 kB 00:00 2024-09-17T16:51:18.976 INFO:teuthology.orchestra.run.smithi037.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-17T16:51:19.001 INFO:teuthology.orchestra.run.smithi037.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-17T16:51:19.027 INFO:teuthology.orchestra.run.smithi037.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-09-17T16:51:19.052 INFO:teuthology.orchestra.run.smithi037.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-17T16:51:19.077 INFO:teuthology.orchestra.run.smithi037.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 658 kB/s | 16 kB 00:00 2024-09-17T16:51:19.102 INFO:teuthology.orchestra.run.smithi037.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-17T16:51:19.128 INFO:teuthology.orchestra.run.smithi037.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T16:51:19.153 INFO:teuthology.orchestra.run.smithi037.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-17T16:51:19.195 INFO:teuthology.orchestra.run.smithi037.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-17T16:51:19.220 INFO:teuthology.orchestra.run.smithi037.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-17T16:51:19.246 INFO:teuthology.orchestra.run.smithi037.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-17T16:51:19.271 INFO:teuthology.orchestra.run.smithi037.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-09-17T16:51:19.274 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:19.274 INFO:teuthology.orchestra.run.smithi037.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-17T16:51:19.432 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:19.477 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:19.477 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:19.991 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:19.991 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:21.389 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:21.457 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T16:51:21.509 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T16:51:21.543 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T16:51:21.620 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T16:51:21.659 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T16:51:21.692 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T16:51:21.791 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T16:51:21.867 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T16:51:21.920 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T16:51:21.955 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T16:51:21.984 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T16:51:22.019 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T16:51:22.052 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T16:51:22.094 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T16:51:22.154 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T16:51:22.195 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T16:51:22.246 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T16:51:22.285 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T16:51:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T16:51:22.534 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T16:51:22.621 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T16:51:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T16:51:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T16:51:22.785 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T16:51:22.859 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T16:51:23.077 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T16:51:23.121 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T16:51:23.203 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T16:51:23.303 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T16:51:23.344 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T16:51:23.482 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T16:51:23.657 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T16:51:23.687 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:51:23.716 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:25.271 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:51:25.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:25.788 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:51:25.814 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:26.064 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T16:51:26.089 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T16:51:26.089 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:26.089 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T16:51:26.089 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:51:26.090 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:51:26.090 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:26.164 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T16:51:27.900 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T16:51:27.901 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T16:51:27.902 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T16:51:27.904 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T16:51:27.905 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T16:51:28.802 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T16:51:28.802 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:28.802 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:28.802 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T16:51:28.803 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T16:51:28.804 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:28.805 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:29.145 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-09-17T16:51:29.751 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:29.844 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:51:29.889 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:29.890 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:51:29.890 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:29.963 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-09-17T16:51:30.571 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:30.708 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:30.709 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:51:30.710 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:30.710 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 784 k 2024-09-17T16:51:30.710 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 791 k 2024-09-17T16:51:30.710 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:30.996 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-17T16:51:30.997 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:30.997 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-17T16:51:30.997 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:31.003 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:31.003 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:31.009 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:31.010 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:31.084 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:31.586 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:51:31.677 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:51:32.087 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:51:32.303 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:51:32.304 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:32.304 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:32.304 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:32.304 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:32.304 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:32.510 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T16:51:33.116 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:33.253 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:33.253 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:33.253 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 145 k 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 432 k 2024-09-17T16:51:33.254 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:33.510 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 567 kB/s | 145 kB 00:00 2024-09-17T16:51:33.510 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:33.511 INFO:teuthology.orchestra.run.smithi037.stdout:Total 564 kB/s | 145 kB 00:00 2024-09-17T16:51:33.511 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:33.519 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:33.519 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:33.581 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:33.581 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:33.731 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:33.794 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T16:51:33.821 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T16:51:33.821 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:51:33.821 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T16:51:33.821 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:34.758 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T16:51:34.759 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:34.759 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:34.759 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:51:34.759 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:34.759 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:34.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-09-17T16:51:35.589 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:35.681 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:51:35.726 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:35.726 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:51:35.727 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:35.804 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T16:51:36.410 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:36.551 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T16:51:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-17T16:51:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:36.555 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 19 M 2024-09-17T16:51:36.555 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 159 M 2024-09-17T16:51:36.555 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:37.005 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4975.g188 108 kB/s | 16 kB 00:00 2024-09-17T16:51:37.030 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4975.g18 193 kB/s | 34 kB 00:00 2024-09-17T16:51:37.172 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-17T16:51:37.222 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-17T16:51:37.281 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T16:51:37.306 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-17T16:51:37.373 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-17T16:51:37.398 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-17T16:51:37.424 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T16:51:37.449 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-17T16:51:37.541 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.3 MB/s | 863 kB 00:00 2024-09-17T16:51:38.958 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 6.7 MB/s | 14 MB 00:02 2024-09-17T16:51:38.962 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:38.962 INFO:teuthology.orchestra.run.smithi037.stdout:Total 8.0 MB/s | 19 MB 00:02 2024-09-17T16:51:39.015 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:39.037 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:39.037 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:39.337 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:39.337 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:39.952 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:40.042 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T16:51:40.077 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T16:51:40.142 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T16:51:40.170 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T16:51:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T16:51:40.428 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T16:51:40.527 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T16:51:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T16:51:40.604 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T16:51:40.636 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T16:51:44.035 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T16:51:44.098 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T16:51:46.028 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T16:51:46.029 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T16:51:46.029 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T16:51:46.029 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T16:51:46.029 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T16:51:46.029 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T16:51:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T16:51:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:46.345 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:46.346 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:46.576 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T16:51:47.186 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:10 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:47.329 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T16:51:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-17T16:51:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:47.332 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 38 M 2024-09-17T16:51:47.332 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 212 M 2024-09-17T16:51:47.332 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:47.688 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-17T16:51:47.805 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-17T16:51:47.831 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-17T16:51:47.881 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-17T16:51:47.990 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T16:51:48.015 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 288 kB/s | 38 kB 00:00 2024-09-17T16:51:48.132 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-17T16:51:48.307 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.0 MB/s | 3.0 MB 00:00 2024-09-17T16:51:48.383 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-09-17T16:51:48.500 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-09-17T16:51:48.641 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.6 MB/s | 4.9 MB 00:00 2024-09-17T16:51:49.117 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-17T16:51:49.118 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:49.118 INFO:teuthology.orchestra.run.smithi037.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-17T16:51:49.482 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:49.509 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:49.509 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:49.951 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:49.952 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:51:50.734 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:51:50.813 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T16:51:50.882 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T16:51:50.987 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T16:51:51.025 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T16:51:51.217 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T16:51:51.248 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T16:51:51.327 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T16:51:52.031 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T16:51:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T16:51:53.484 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T16:51:54.577 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T16:51:54.605 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T16:51:57.213 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T16:51:57.214 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T16:51:57.214 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T16:51:57.214 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T16:51:57.214 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T16:51:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T16:51:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:57.614 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:51:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:57.615 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:51:58.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-09-17T16:51:58.627 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:21 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:51:58.765 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T16:51:58.767 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-09-17T16:51:58.768 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:51:58.769 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.2 M 2024-09-17T16:51:58.769 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-09-17T16:51:58.769 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:51:58.989 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 225 kB/s | 19 kB 00:00 2024-09-17T16:51:59.006 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 251 kB/s | 26 kB 00:00 2024-09-17T16:51:59.073 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 292 kB/s | 49 kB 00:00 2024-09-17T16:51:59.098 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-17T16:51:59.132 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-17T16:51:59.157 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-17T16:51:59.207 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-17T16:51:59.233 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-09-17T16:51:59.258 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-17T16:51:59.291 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-17T16:51:59.325 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-09-17T16:51:59.350 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-17T16:51:59.392 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-17T16:51:59.395 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:51:59.395 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-17T16:51:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:51:59.477 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:51:59.477 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:51:59.574 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:51:59.576 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:00.116 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:00.302 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T16:52:00.371 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T16:52:00.427 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T16:52:00.470 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T16:52:00.512 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T16:52:00.620 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T16:52:00.730 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T16:52:00.781 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T16:52:01.564 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T16:52:01.627 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T16:52:01.660 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T16:52:01.710 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T16:52:01.738 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T16:52:02.514 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T16:52:02.515 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T16:52:02.515 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T16:52:02.515 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T16:52:02.515 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T16:52:02.515 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T16:52:02.870 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T16:52:02.870 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:02.870 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:02.870 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:52:02.870 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:02.871 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:03.107 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T16:52:03.708 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:26 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:03.848 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T16:52:03.849 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-09-17T16:52:03.850 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:03.851 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 7.0 M 2024-09-17T16:52:03.851 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-09-17T16:52:03.851 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:04.352 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 773 kB/s | 154 kB 00:00 2024-09-17T16:52:04.411 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T16:52:04.436 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T16:52:04.461 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-17T16:52:04.978 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 301 kB/s | 248 kB 00:00 2024-09-17T16:52:05.345 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-09-17T16:52:05.347 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:05.348 INFO:teuthology.orchestra.run.smithi037.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-17T16:52:05.424 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:05.437 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:05.438 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:05.545 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:05.857 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:06.028 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T16:52:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T16:52:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T16:52:06.737 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T16:52:06.821 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T16:52:06.850 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T16:52:07.672 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T16:52:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T16:52:07.952 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T16:52:07.952 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:07.952 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:08.204 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-09-17T16:52:08.813 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:31 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:08.952 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T16:52:08.953 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 979 k 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.9 M 2024-09-17T16:52:08.954 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:09.230 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 533 kB/s | 80 kB 00:00 2024-09-17T16:52:09.730 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 1.4 MB/s | 899 kB 00:00 2024-09-17T16:52:09.730 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:09.731 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.2 MB/s | 979 kB 00:00 2024-09-17T16:52:09.745 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:09.771 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:09.772 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:09.862 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:09.862 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:10.056 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:10.203 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T16:52:10.283 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T16:52:10.989 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T16:52:10.989 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T16:52:11.249 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T16:52:11.249 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:11.250 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:11.250 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T16:52:11.250 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:11.250 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:11.512 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-volume 2024-09-17T16:52:12.121 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:12.261 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:12.261 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:12.261 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-17T16:52:12.261 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 285 k 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.3 M 2024-09-17T16:52:12.262 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:12.634 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 766 kB/s | 285 kB 00:00 2024-09-17T16:52:12.634 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:12.635 INFO:teuthology.orchestra.run.smithi037.stdout:Total 763 kB/s | 285 kB 00:00 2024-09-17T16:52:12.635 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:12.644 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:12.644 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:12.671 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:12.671 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:12.902 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:52:12.994 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:52:12.994 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:52:12.994 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T16:52:12.994 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:13.974 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T16:52:13.974 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:13.975 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:13.975 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T16:52:13.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:13.975 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:14.257 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-09-17T16:52:14.870 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:15.008 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:15.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:15.010 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 126 k 2024-09-17T16:52:15.010 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-09-17T16:52:15.010 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:15.230 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 575 kB/s | 126 kB 00:00 2024-09-17T16:52:15.231 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:15.231 INFO:teuthology.orchestra.run.smithi037.stdout:Total 571 kB/s | 126 kB 00:00 2024-09-17T16:52:15.231 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:15.237 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:15.237 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:15.306 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:15.306 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:15.469 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:15.566 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:16.009 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:16.265 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:52:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:16.265 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:16.492 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-09-17T16:52:17.118 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:17.215 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:17.262 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:17.263 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:17.263 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:17.333 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-09-17T16:52:17.947 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:18.085 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 32 k 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 142 k 2024-09-17T16:52:18.087 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:18.242 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 204 kB/s | 32 kB 00:00 2024-09-17T16:52:18.243 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:18.243 INFO:teuthology.orchestra.run.smithi037.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-17T16:52:18.243 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:18.247 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:18.247 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:18.270 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:18.270 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:18.364 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:18.890 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:19.131 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:19.316 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-09-17T16:52:19.920 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:20.013 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:20.061 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:20.062 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:20.062 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:20.133 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-09-17T16:52:20.735 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:20.828 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:20.873 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:20.874 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:20.951 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-09-17T16:52:21.556 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:21.648 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:21.695 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:21.696 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:21.696 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:21.768 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-09-17T16:52:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:22.467 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:22.513 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:22.514 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:22.589 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-09-17T16:52:23.202 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:23.295 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:23.341 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:23.342 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:23.417 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-09-17T16:52:24.019 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:24.113 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T16:52:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-17T16:52:24.160 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:24.356 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-09-17T16:52:24.962 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:25.101 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:25.102 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 85 k 2024-09-17T16:52:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 231 k 2024-09-17T16:52:25.102 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:25.346 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 350 kB/s | 85 kB 00:00 2024-09-17T16:52:25.346 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:25.346 INFO:teuthology.orchestra.run.smithi037.stdout:Total 347 kB/s | 85 kB 00:00 2024-09-17T16:52:25.346 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:25.354 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:25.354 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:25.422 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:25.422 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:25.688 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:25.787 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:26.213 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:26.481 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:26.481 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:26.481 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:26.481 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:52:26.482 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:26.482 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:26.789 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-09-17T16:52:27.404 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:27.546 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.2 M 2024-09-17T16:52:27.547 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 14 M 2024-09-17T16:52:27.548 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:27.865 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-17T16:52:27.867 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:27.867 INFO:teuthology.orchestra.run.smithi037.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-17T16:52:27.867 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:27.881 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:27.881 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:27.967 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:27.967 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:28.436 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:28.558 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:28.585 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:28.585 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:52:28.586 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T16:52:28.586 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:52:28.586 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:52:28.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:29.760 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:29.760 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:29.761 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:29.761 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:52:29.761 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:29.761 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:30.012 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-09-17T16:52:30.624 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:53 ago on Tue 17 Sep 2024 04:50:37 PM UTC. 2024-09-17T16:52:30.765 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:30.766 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 170 k 2024-09-17T16:52:30.767 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 490 k 2024-09-17T16:52:30.767 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-17T16:52:30.977 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 809 kB/s | 170 kB 00:00 2024-09-17T16:52:30.978 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:52:30.978 INFO:teuthology.orchestra.run.smithi037.stdout:Total 803 kB/s | 170 kB 00:00 2024-09-17T16:52:30.978 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-17T16:52:30.986 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-17T16:52:30.986 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-17T16:52:31.054 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-17T16:52:31.054 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-17T16:52:31.214 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-17T16:52:31.319 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-17T16:52:32.015 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-17T16:52:32.276 DEBUG:teuthology.parallel:result is None 2024-09-17T16:52:32.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:52:32.420 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T16:52:32.442 INFO:teuthology.orchestra.run.smithi037.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T16:52:32.443 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T16:52:32.443 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T16:52:32.445 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T16:52:32.445 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:52:32.445 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T16:52:32.508 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T16:52:32.509 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:52:32.509 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T16:52:32.576 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T16:52:32.641 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T16:52:32.642 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:52:32.642 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T16:52:32.706 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T16:52:32.771 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T16:52:32.771 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:52:32.772 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T16:52:32.836 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T16:52:32.902 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T16:52:32.916 INFO:tasks.workunit:Pulling workunits from ref 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:52:32.916 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T16:52:32.917 INFO:tasks.workunit:timeout=3h 2024-09-17T16:52:32.917 INFO:tasks.workunit:cleanup=True 2024-09-17T16:52:32.917 DEBUG:teuthology.orchestra.run.smithi037:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T16:52:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:52:32.958 INFO:teuthology.orchestra.run.smithi037.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T16:52:32.958 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T16:52:33.014 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T16:52:33.014 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T16:52:33.070 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:52:33.127 INFO:tasks.workunit.client.0.smithi037.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T16:53:22.991 INFO:tasks.workunit.client.0.smithi037.stderr:Updating files: 97% (11780/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:Note: switching to '1889a165b5128787ec693450a3d1c5df5d6dc203'. 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T16:53:23.508 INFO:tasks.workunit.client.0.smithi037.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: git switch -c 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr:Or undo this operation with: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: git switch - 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-17T16:53:23.509 INFO:tasks.workunit.client.0.smithi037.stderr:HEAD is now at 1889a165b51 Merge branch 'wip-matanb-cleanup-snapmap' of https://github.com/Matan-B/ceph into wip-bharath10-testing-2024-09-17-1348 2024-09-17T16:53:23.514 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T16:53:23.570 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-17T16:53:23.571 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T16:53:23.627 INFO:tasks.workunit:Running workunits matching mon-stretch on client.0... 2024-09-17T16:53:23.627 INFO:tasks.workunit:Running workunit mon-stretch/mon-stretch-fail-recovery.sh... 2024-09-17T16:53:23.628 DEBUG:teuthology.orchestra.run.smithi037:workunit test mon-stretch/mon-stretch-fail-recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh 2024-09-17T16:53:23.699 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T16:53:23.705 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T16:53:23.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:53:23.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T16:53:23.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T16:53:23.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/mon-stretch-fail-recovery 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:5: run: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:6: run: shift 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:8: run: export CEPH_MON_A=127.0.0.1:7139 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:8: run: CEPH_MON_A=127.0.0.1:7139 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:9: run: export CEPH_MON_B=127.0.0.1:7141 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:9: run: CEPH_MON_B=127.0.0.1:7141 2024-09-17T16:53:23.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:10: run: export CEPH_MON_C=127.0.0.1:7142 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:10: run: CEPH_MON_C=127.0.0.1:7142 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:11: run: export CEPH_MON_D=127.0.0.1:7143 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:11: run: CEPH_MON_D=127.0.0.1:7143 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:12: run: export CEPH_MON_E=127.0.0.1:7144 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:12: run: CEPH_MON_E=127.0.0.1:7144 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:13: run: export CEPH_ARGS 2024-09-17T16:53:23.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:14: run: uuidgen 2024-09-17T16:53:23.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:14: run: CEPH_ARGS+='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none ' 2024-09-17T16:53:23.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:16: run: export 'BASE_CEPH_ARGS=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none ' 2024-09-17T16:53:23.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:16: run: BASE_CEPH_ARGS='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none ' 2024-09-17T16:53:23.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:17: run: CEPH_ARGS+=--mon-host=127.0.0.1:7139 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: set 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: local funcs=TEST_stretched_cluster_failover_add_three_osds 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:20: run: for func in $funcs 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:21: run: setup td/mon-stretch-fail-recovery 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretch-fail-recovery 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:53:23.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:53:23.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:53:23.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:53:23.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:53:23.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:53:23.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:53:23.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:53:23.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:53:23.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:53:23.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:53:23.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:53:23.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:53:23.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:53:23.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:53:23.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:53:23.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:53:23.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2024-09-17T16:53:23.724 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:53:23.724 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:23.724 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:23.724 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27368 2024-09-17T16:53:23.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:53:23.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:53:23.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretch-fail-recovery 2024-09-17T16:53:23.726 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:53:23.726 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:23.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:23.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27368 2024-09-17T16:53:23.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:53:23.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T16:53:23.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretch-fail-recovery 1' TERM HUP INT 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:22: run: TEST_stretched_cluster_failover_add_three_osds td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:27: TEST_stretched_cluster_failover_add_three_osds: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:28: TEST_stretched_cluster_failover_add_three_osds: local OSDS=8 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:29: TEST_stretched_cluster_failover_add_three_osds: setup td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T16:53:23.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T16:53:23.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T16:53:23.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T16:53:23.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T16:53:23.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T16:53:23.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T16:53:23.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T16:53:23.732 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T16:53:23.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T16:53:23.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T16:53:23.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T16:53:23.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:53:23.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T16:53:23.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T16:53:23.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T16:53:23.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:53:23.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T16:53:23.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2024-09-17T16:53:23.736 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T16:53:23.737 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:23.737 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:23.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27368 2024-09-17T16:53:23.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T16:53:23.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T16:53:23.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretch-fail-recovery 2024-09-17T16:53:23.739 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T16:53:23.739 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:23.739 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:23.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27368 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretch-fail-recovery 1' TERM HUP INT 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:31: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery a --public-addr 127.0.0.1:7139 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:23.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:53:23.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T16:53:23.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:53:23.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/a 2024-09-17T16:53:23.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mon-stretch-fail-recovery/a --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7139 2024-09-17T16:53:24.146 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:53:24.146 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:24.146 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:24.146 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:24.146 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:24.147 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:24.147 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:53:24.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/a '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7139 2024-09-17T16:53:24.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:53:24.341 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.a.asok 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:24.342 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.a.asok config get fsid 2024-09-17T16:53:24.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T16:53:24.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:24.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T16:53:24.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:53:24.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:53:24.445 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T16:53:24.445 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T16:53:24.445 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T16:53:24.445 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:24.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:24.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:24.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.a.asok 2024-09-17T16:53:24.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:24.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.a.asok config get mon_host 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:32: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 1 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=1 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 1 ]] 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T16:53:24.527 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T16:53:24.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592304 2024-09-17T16:53:24.528 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T16:53:24.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592004 -lt 1726592304 ]] 2024-09-17T16:53:24.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 1' 2024-09-17T16:53:24.529 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T16:53:25.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T16:53:25.333 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:23.761820Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:25.334 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:23.761820Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:25.334 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 1' 2024-09-17T16:53:25.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T16:53:25.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:34: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery b --public-addr 127.0.0.1:7141 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-17T16:53:25.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:53:25.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/b 2024-09-17T16:53:25.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/mon-stretch-fail-recovery/b --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7141 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:53:25.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/b '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7141 2024-09-17T16:53:26.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:53:26.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-17T16:53:26.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:26.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-17T16:53:26.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.b.asok 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:26.007 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.b.asok config get fsid 2024-09-17T16:53:26.095 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-17T16:53:26.095 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.b.asok 2024-09-17T16:53:26.096 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:26.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.b.asok config get mon_host 2024-09-17T16:53:26.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:35: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141' 2024-09-17T16:53:26.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:36: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 2 2024-09-17T16:53:26.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T16:53:26.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2024-09-17T16:53:26.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T16:53:26.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2024-09-17T16:53:26.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T16:53:26.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T16:53:26.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592306 2024-09-17T16:53:26.179 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T16:53:26.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592006 -lt 1726592306 ]] 2024-09-17T16:53:26.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-17T16:53:26.180 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T16:53:35.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T16:53:35.383 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":2,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:26.004410Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:35.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":2,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:26.004410Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:35.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-17T16:53:35.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T16:53:35.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:38: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery c --public-addr 127.0.0.1:7142 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/c 2024-09-17T16:53:35.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/mon-stretch-fail-recovery/c --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7142 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:35.961 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:35.962 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:53:35.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/c '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7142 2024-09-17T16:53:36.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:53:36.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-17T16:53:36.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:36.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-17T16:53:36.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:36.215 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:36.216 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.c.asok 2024-09-17T16:53:36.216 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:36.216 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.c.asok config get fsid 2024-09-17T16:53:36.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-17T16:53:36.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:36.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-17T16:53:36.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.c.asok 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:36.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.c.asok config get mon_host 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:39: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142' 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:40: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 3 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T16:53:36.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T16:53:36.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592316 2024-09-17T16:53:36.392 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T16:53:36.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592016 -lt 1726592316 ]] 2024-09-17T16:53:36.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-17T16:53:36.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T16:53:42.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T16:53:42.671 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":3,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:36.341532Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:42.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":3,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:36.341532Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:42.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-17T16:53:42.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T16:53:42.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T16:53:42.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T16:53:42.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:42: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery d --public-addr 127.0.0.1:7143 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/d 2024-09-17T16:53:42.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/mon-stretch-fail-recovery/d --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7143 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:43.255 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:53:43.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/d '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7143 2024-09-17T16:53:43.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:53:43.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-17T16:53:43.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:43.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-17T16:53:43.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:53:43.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:43.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.d.asok 2024-09-17T16:53:43.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:43.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.d.asok config get fsid 2024-09-17T16:53:43.545 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.d.asok 2024-09-17T16:53:43.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:43.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.d.asok config get mon_host 2024-09-17T16:53:43.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:43: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143' 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:44: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 4 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T16:53:43.629 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T16:53:43.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592323 2024-09-17T16:53:43.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T16:53:43.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592023 -lt 1726592323 ]] 2024-09-17T16:53:43.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-17T16:53:43.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T16:53:49.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T16:53:49.836 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":4,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:43.602638Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:49.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":4,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:43.602638Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:49.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T16:53:49.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:46: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery e --public-addr 127.0.0.1:7144 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/e 2024-09-17T16:53:49.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/mon-stretch-fail-recovery/e --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7144 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:53:50.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/e '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7144 2024-09-17T16:53:50.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T16:53:50.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2024-09-17T16:53:50.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:50.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-17T16:53:50.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T16:53:50.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T16:53:50.672 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-17T16:53:50.672 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.e.asok 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:50.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.e.asok config get fsid 2024-09-17T16:53:50.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2024-09-17T16:53:50.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T16:53:50.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-17T16:53:50.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T16:53:50.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T16:53:50.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-17T16:53:50.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-17T16:53:50.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-17T16:53:50.767 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T16:53:50.768 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:53:50.768 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:53:50.768 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27368/ceph-mon.e.asok 2024-09-17T16:53:50.768 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T16:53:50.768 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27368/ceph-mon.e.asok config get mon_host 2024-09-17T16:53:50.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:47: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:48: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 5 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T16:53:50.850 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T16:53:50.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592330 2024-09-17T16:53:50.852 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T16:53:50.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592030 -lt 1726592330 ]] 2024-09-17T16:53:50.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-17T16:53:50.852 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T16:53:57.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T16:53:57.529 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":5,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:50.815600Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:57.529 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":5,"fsid":"bcd61817-a54f-4023-86d3-fc20ff4fbf27","modified":"2024-09-17T16:53:50.815600Z","created":"2024-09-17T16:53:23.761820Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T16:53:57.529 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2024-09-17T16:53:57.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T16:53:57.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T16:53:57.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T16:53:57.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T16:53:57.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T16:53:57.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:50: TEST_stretched_cluster_failover_add_three_osds: ceph mon set election_strategy connectivity 2024-09-17T16:54:03.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:51: TEST_stretched_cluster_failover_add_three_osds: ceph mon add disallowed_leader e 2024-09-17T16:54:04.689 INFO:tasks.workunit.client.0.smithi037.stderr:mon.e is already disallowed 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:53: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery x 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/x 2024-09-17T16:54:04.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:54:05.245 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:54:05.245 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:05.245 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:05.245 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:05.245 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:05.246 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:54:05.246 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:05.246 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:05.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/x '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:54: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery y 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=y 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/y 2024-09-17T16:54:08.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:54:08.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:54:08.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:08.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/y '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:55: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery z 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=z 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/z 2024-09-17T16:54:11.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T16:54:11.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T16:54:11.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:11.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:11.735 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:11.736 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:11.736 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:54:11.736 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:11.736 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:11.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id z --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/z '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T16:54:14.765 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: expr 8 - 1 2024-09-17T16:54:14.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: seq 0 7 2024-09-17T16:54:14.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 0 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/0 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:54:14.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/0' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/0/journal' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:14.823 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:54:14.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:54:14.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:54:14.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/0 2024-09-17T16:54:14.826 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:54:14.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=95ac4b42-1f8d-4f91-8569-9413f9333c01 2024-09-17T16:54:14.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 95ac4b42-1f8d-4f91-8569-9413f9333c01' 2024-09-17T16:54:14.827 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 95ac4b42-1f8d-4f91-8569-9413f9333c01 2024-09-17T16:54:14.827 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:54:14.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2tOlm6WrwMRAAI9e8mchNsLOr45kO+Il3wQ== 2024-09-17T16:54:14.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2tOlm6WrwMRAAI9e8mchNsLOr45kO+Il3wQ=="}' 2024-09-17T16:54:14.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 95ac4b42-1f8d-4f91-8569-9413f9333c01 -i td/mon-stretch-fail-recovery/0/new.json 2024-09-17T16:54:15.746 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:54:15.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/0/new.json 2024-09-17T16:54:15.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/0 --osd-journal=td/mon-stretch-fail-recovery/0/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2tOlm6WrwMRAAI9e8mchNsLOr45kO+Il3wQ== --osd-uuid 95ac4b42-1f8d-4f91-8569-9413f9333c01 2024-09-17T16:54:15.877 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:15.875+0000 7efc725c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:15.879 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:15.877+0000 7efc725c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:15.881 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:15.878+0000 7efc725c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:15.881 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:15.879+0000 7efc725c5640 -1 bdev(0x56125588bc00 td/mon-stretch-fail-recovery/0/block) open stat got: (1) Operation not permitted 2024-09-17T16:54:15.881 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:15.879+0000 7efc725c5640 -1 bluestore(td/mon-stretch-fail-recovery/0) _read_fsid unparsable uuid 2024-09-17T16:54:22.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/0/keyring 2024-09-17T16:54:22.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:54:22.327 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-17T16:54:22.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T16:54:22.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:54:23.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T16:54:23.025 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-17T16:54:23.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/0 --osd-journal=td/mon-stretch-fail-recovery/0/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:54:23.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:54:23.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:54:23.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:54:23.052 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:23.050+0000 7fab0cda6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:23.054 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:23.052+0000 7fab0cda6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:23.055 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:23.053+0000 7fab0cda6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:23.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:54:23.656 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:54:23.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:23.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:24.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:24.407 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:24.405+0000 7fab0cda6640 -1 Falling back to public interface 2024-09-17T16:54:25.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:25.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:25.303 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:54:25.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:54:25.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:25.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:25.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:26.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:26.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:26.721 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:54:26.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:54:26.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:26.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:27.102 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:27.099+0000 7fab0cda6640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:54:27.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:28.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:28.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:28.406 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:54:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:54:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:29.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:30.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:30.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:30.450 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:54:30.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:54:30.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:30.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:31.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:32.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:32.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:32.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:54:32.318 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:54:32.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:32.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:33.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:34.155 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:34.155 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:34.155 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:54:34.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:54:34.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:34.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:34.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:35.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:35.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:35.713 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:54:35.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:54:35.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:35.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:36.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:37.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:37.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:37.517 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:54:37.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:54:37.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:37.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:38.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:39.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:39.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:39.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:54:39.557 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:54:39.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:39.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:40.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:41.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:41.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:41.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:54:41.609 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:54:41.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:41.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:42.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:43.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:43.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:43.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:54:43.374 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:54:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:45.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:54:46.607 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:54:46.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:46.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:47.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:48.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:48.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:48.597 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:54:48.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:54:48.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:48.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:51.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:52.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:52.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:52.312 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:54:52.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:54:52.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:52.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:54.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:54:55.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:54:55.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:54:55.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:54:55.762 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T16:54:55.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:54:55.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T16:54:56.661 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2378995805,v1:127.0.0.1:6803/2378995805] [v2:127.0.0.1:6804/2378995805,v1:127.0.0.1:6805/2378995805] exists,up 95ac4b42-1f8d-4f91-8569-9413f9333c01 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 1 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:54:56.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/1 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/1' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/1/journal' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:54:56.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:56.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:54:56.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/1 2024-09-17T16:54:56.669 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:54:56.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a 2024-09-17T16:54:56.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a' 2024-09-17T16:54:56.669 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a 2024-09-17T16:54:56.670 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:54:56.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBgtOlmzyOeKBAAcbuooT4nk8VlKT8jM3G5aQ== 2024-09-17T16:54:56.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBgtOlmzyOeKBAAcbuooT4nk8VlKT8jM3G5aQ=="}' 2024-09-17T16:54:56.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a -i td/mon-stretch-fail-recovery/1/new.json 2024-09-17T16:54:57.395 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:54:57.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/1/new.json 2024-09-17T16:54:57.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/1 --osd-journal=td/mon-stretch-fail-recovery/1/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgtOlmzyOeKBAAcbuooT4nk8VlKT8jM3G5aQ== --osd-uuid e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a 2024-09-17T16:54:57.436 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:57.435+0000 7f7317216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:57.462 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:57.460+0000 7f7317216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:57.463 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:57.462+0000 7f7317216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:54:57.463 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:57.462+0000 7f7317216640 -1 bdev(0x55a1ce0ddc00 td/mon-stretch-fail-recovery/1/block) open stat got: (1) Operation not permitted 2024-09-17T16:54:57.464 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:54:57.462+0000 7f7317216640 -1 bluestore(td/mon-stretch-fail-recovery/1) _read_fsid unparsable uuid 2024-09-17T16:55:04.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/1/keyring 2024-09-17T16:55:04.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:55:04.041 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-17T16:55:04.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T16:55:04.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:55:04.736 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-17T16:55:04.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T16:55:04.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/1 --osd-journal=td/mon-stretch-fail-recovery/1/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:55:04.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:55:04.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:55:04.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:55:04.762 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:04.760+0000 7f2c61c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:04.764 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:04.763+0000 7f2c61c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:04.765 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:04.764+0000 7f2c61c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:05.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T16:55:05.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:55:05.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T16:55:05.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:55:05.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:55:05.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:05.359 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:55:05.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:55:05.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:05.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:06.113 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:06.112+0000 7f2c61c15640 -1 Falling back to public interface 2024-09-17T16:55:06.145 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:07.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:07.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:07.147 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:55:07.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:55:07.148 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:07.148 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:08.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:08.905 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:08.903+0000 7f2c61c15640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:55:09.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:09.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:09.120 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:55:09.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:09.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:09.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:09.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:10.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:10.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:10.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:10.826 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:55:10.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:10.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:11.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:12.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:12.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:12.590 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:55:12.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:55:12.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:12.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:13.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:14.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:14.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:14.731 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:55:14.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:55:14.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:14.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:15.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:16.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:16.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:16.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:55:16.357 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:55:16.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:16.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:17.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:18.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:18.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:18.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:55:18.513 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:55:18.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:18.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:19.455 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:20.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:20.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:20.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:55:20.457 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:55:20.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:20.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:20.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:21.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:21.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:21.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:55:21.964 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:55:21.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:21.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:23.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:24.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:24.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:24.199 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:55:24.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:55:24.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:24.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:26.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:26.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:26.087 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:55:26.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:55:26.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:26.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:26.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:27.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:27.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:27.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:55:27.759 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:55:27.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:27.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:29.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:30.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:30.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:30.069 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:55:30.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:55:30.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:30.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:30.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:31.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:31.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:31.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:55:31.954 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:55:31.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:31.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:33.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:34.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:34.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:34.845 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T16:55:34.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:55:34.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:34.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:35.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:36.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:36.483 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-17T16:55:36.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:36.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:55:36.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:36.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:37.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:38.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:38.970 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-17T16:55:38.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:38.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T16:55:38.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:38.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T16:55:39.803 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/532143347,v1:127.0.0.1:6811/532143347] [v2:127.0.0.1:6812/532143347,v1:127.0.0.1:6813/532143347] exists,up e9aab1d2-6a4f-4551-86f3-ab0b37a6a91a 2024-09-17T16:55:39.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:55:39.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:55:39.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 2 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/2 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/2' 2024-09-17T16:55:39.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/2/journal' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:55:39.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/2 2024-09-17T16:55:39.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:55:39.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=53af092b-ae8c-4d7f-97be-d17e2da93912 2024-09-17T16:55:39.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 53af092b-ae8c-4d7f-97be-d17e2da93912' 2024-09-17T16:55:39.810 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 53af092b-ae8c-4d7f-97be-d17e2da93912 2024-09-17T16:55:39.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:55:39.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLtOlmxDgYMRAA/VDbjRuxYhG3xQdPHHz/EQ== 2024-09-17T16:55:39.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLtOlmxDgYMRAA/VDbjRuxYhG3xQdPHHz/EQ=="}' 2024-09-17T16:55:39.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 53af092b-ae8c-4d7f-97be-d17e2da93912 -i td/mon-stretch-fail-recovery/2/new.json 2024-09-17T16:55:40.766 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:55:40.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/2/new.json 2024-09-17T16:55:40.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/2 --osd-journal=td/mon-stretch-fail-recovery/2/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLtOlmxDgYMRAA/VDbjRuxYhG3xQdPHHz/EQ== --osd-uuid 53af092b-ae8c-4d7f-97be-d17e2da93912 2024-09-17T16:55:40.806 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:40.804+0000 7fd46620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:40.808 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:40.806+0000 7fd46620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:40.809 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:40.807+0000 7fd46620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:40.810 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:40.807+0000 7fd46620f640 -1 bdev(0x55de7ea7bc00 td/mon-stretch-fail-recovery/2/block) open stat got: (1) Operation not permitted 2024-09-17T16:55:40.810 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:40.807+0000 7fd46620f640 -1 bluestore(td/mon-stretch-fail-recovery/2) _read_fsid unparsable uuid 2024-09-17T16:55:47.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/2/keyring 2024-09-17T16:55:47.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:55:47.227 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-17T16:55:47.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T16:55:47.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:55:48.079 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-17T16:55:48.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T16:55:48.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/2 --osd-journal=td/mon-stretch-fail-recovery/2/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:55:48.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:55:48.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:55:48.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:55:48.105 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:48.102+0000 7fd49863b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:48.106 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:48.104+0000 7fd49863b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:48.107 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:48.105+0000 7fd49863b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:55:48.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:55:48.750 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:55:48.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:48.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:49.198 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:49.196+0000 7fd49863b640 -1 Falling back to public interface 2024-09-17T16:55:49.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:50.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:50.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:50.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:55:50.398 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:55:50.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:50.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:50.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:51.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:51.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:51.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:55:51.974 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:55:51.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:51.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:52.139 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:55:52.137+0000 7fd49863b640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:55:52.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:53.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:53.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:53.723 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:55:53.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:55:53.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:53.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:54.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:55.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:55.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:55.930 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:55:55.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:55:55.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:55.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:56.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:57.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:57.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:57.517 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:55:57.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:55:57.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:57.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:55:58.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:55:59.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:55:59.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:55:59.819 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:55:59.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:55:59.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:55:59.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:00.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:01.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:01.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:01.378 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:56:01.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:56:01.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:01.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:02.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:03.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:03.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:03.421 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:56:03.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:56:03.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:03.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:04.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:05.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:05.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:05.568 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:56:05.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:56:05.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:05.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:06.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:07.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:07.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:07.182 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:56:07.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:56:07.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:07.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:08.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:09.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:09.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:09.017 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:56:09.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:56:09.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:09.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:10.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:11.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:11.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:11.182 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:56:11.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:56:11.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:11.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:12.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:13.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:13.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:13.185 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:56:13.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:56:13.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:13.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:15.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:16.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:16.380 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:56:16.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:16.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:56:16.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:16.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:17.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:18.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:18.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:18.771 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T16:56:18.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:56:18.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:18.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:19.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:20.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:20.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:20.915 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-17T16:56:20.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T16:56:20.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:20.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T16:56:21.368 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1768369849,v1:127.0.0.1:6819/1768369849] [v2:127.0.0.1:6820/1768369849,v1:127.0.0.1:6821/1768369849] exists,up 53af092b-ae8c-4d7f-97be-d17e2da93912 2024-09-17T16:56:21.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:56:21.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 3 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/3 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:56:21.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/3' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/3/journal' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:56:21.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/3 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:56:21.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eb7a017e-cb3d-4d44-a12d-4ba52f36df28 2024-09-17T16:56:21.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 eb7a017e-cb3d-4d44-a12d-4ba52f36df28' 2024-09-17T16:56:21.376 INFO:tasks.workunit.client.0.smithi037.stdout:add osd3 eb7a017e-cb3d-4d44-a12d-4ba52f36df28 2024-09-17T16:56:21.377 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:56:21.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1tOlmPOcbFxAARCHe25y+Je6WsVYx05FVbA== 2024-09-17T16:56:21.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1tOlmPOcbFxAARCHe25y+Je6WsVYx05FVbA=="}' 2024-09-17T16:56:21.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eb7a017e-cb3d-4d44-a12d-4ba52f36df28 -i td/mon-stretch-fail-recovery/3/new.json 2024-09-17T16:56:23.064 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:56:23.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/3/new.json 2024-09-17T16:56:23.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/3 --osd-journal=td/mon-stretch-fail-recovery/3/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1tOlmPOcbFxAARCHe25y+Je6WsVYx05FVbA== --osd-uuid eb7a017e-cb3d-4d44-a12d-4ba52f36df28 2024-09-17T16:56:23.111 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:23.110+0000 7fab9b8e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:23.157 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:23.156+0000 7fab9b8e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:23.158 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:23.157+0000 7fab9b8e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:23.159 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:23.157+0000 7fab9b8e0640 -1 bdev(0x55d5502f7c00 td/mon-stretch-fail-recovery/3/block) open stat got: (1) Operation not permitted 2024-09-17T16:56:23.159 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:23.157+0000 7fab9b8e0640 -1 bluestore(td/mon-stretch-fail-recovery/3) _read_fsid unparsable uuid 2024-09-17T16:56:29.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/3/keyring 2024-09-17T16:56:29.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:56:29.746 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd3 key to auth repository 2024-09-17T16:56:29.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T16:56:29.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:56:30.456 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.3 2024-09-17T16:56:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T16:56:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/3 --osd-journal=td/mon-stretch-fail-recovery/3/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:56:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:56:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:56:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:56:30.482 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:30.480+0000 7f2f4ce24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:30.484 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:30.483+0000 7f2f4ce24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:30.485 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:30.484+0000 7f2f4ce24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:56:30.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:30.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:56:30.901 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:56:30.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:30.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:31.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:31.843+0000 7f2f4ce24640 -1 Falling back to public interface 2024-09-17T16:56:32.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:32.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:32.570 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:56:32.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:56:32.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:32.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:33.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:34.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:34.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:34.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:56:34.190 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:56:34.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:34.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:34.523 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:56:34.521+0000 7f2f4ce24640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:56:34.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:35.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:35.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:35.859 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:56:35.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:56:35.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:35.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:36.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:37.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:37.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:37.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:56:37.964 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:56:37.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:37.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:39.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:40.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:40.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:40.612 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:56:40.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:56:40.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:40.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:41.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:42.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:42.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:42.324 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:56:42.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:56:42.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:42.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:43.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:44.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:44.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:44.269 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:56:44.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:56:44.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:44.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:44.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:45.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:45.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:45.796 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:56:45.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:56:45.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:45.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:46.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:47.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:47.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:47.889 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:56:47.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:56:47.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:47.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:48.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:49.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:49.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:49.927 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:56:49.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:56:49.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:49.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:50.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:51.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:51.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:51.619 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:56:51.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:56:51.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:51.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:56:56.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:56:57.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:56:57.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:56:57.996 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:56:57.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:56:57.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:56:57.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:57:00.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:01.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:01.229 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:57:01.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:01.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:57:01.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:01.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:57:03.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:04.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:04.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:04.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:57:04.212 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:57:04.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:04.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T16:57:05.114 INFO:tasks.workunit.client.0.smithi037.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2374578199,v1:127.0.0.1:6827/2374578199] [v2:127.0.0.1:6828/2374578199,v1:127.0.0.1:6829/2374578199] exists,up eb7a017e-cb3d-4d44-a12d-4ba52f36df28 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 4 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/4 2024-09-17T16:57:05.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/4' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/4/journal' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:57:05.119 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:57:05.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/4 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:57:05.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d 2024-09-17T16:57:05.122 INFO:tasks.workunit.client.0.smithi037.stdout:add osd4 22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d 2024-09-17T16:57:05.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d' 2024-09-17T16:57:05.122 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:57:05.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhtOlmem36BxAAWTPoHvQPpLFvSfJhSaGmXg== 2024-09-17T16:57:05.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhtOlmem36BxAAWTPoHvQPpLFvSfJhSaGmXg=="}' 2024-09-17T16:57:05.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d -i td/mon-stretch-fail-recovery/4/new.json 2024-09-17T16:57:06.210 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:57:06.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/4/new.json 2024-09-17T16:57:06.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/4 --osd-journal=td/mon-stretch-fail-recovery/4/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhtOlmem36BxAAWTPoHvQPpLFvSfJhSaGmXg== --osd-uuid 22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d 2024-09-17T16:57:06.251 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:06.249+0000 7ff90c016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:09.255 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:09.252+0000 7ff90c016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:09.256 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:09.253+0000 7ff90c016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:09.256 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:09.254+0000 7ff90c016640 -1 bdev(0x55a4b9bb6800 td/mon-stretch-fail-recovery/4/block) open stat got: (1) Operation not permitted 2024-09-17T16:57:09.257 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:09.254+0000 7ff90c016640 -1 bluestore(td/mon-stretch-fail-recovery/4) _read_fsid unparsable uuid 2024-09-17T16:57:15.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/4/keyring 2024-09-17T16:57:15.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:57:15.996 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd4 key to auth repository 2024-09-17T16:57:15.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T16:57:15.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:57:16.738 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.4 2024-09-17T16:57:16.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T16:57:16.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/4 --osd-journal=td/mon-stretch-fail-recovery/4/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:57:16.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:57:16.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:57:16.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:16.762+0000 7f3621c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:16.766 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:16.764+0000 7f3621c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:16.767 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:16.765+0000 7f3621c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:57:17.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:17.347 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:57:17.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:57:17.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:17.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:17.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:18.121 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:18.118+0000 7f3621c12640 -1 Falling back to public interface 2024-09-17T16:57:18.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:18.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:18.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:57:18.816 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:57:18.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:18.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:20.760 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:20.757+0000 7f3621c12640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:57:22.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:23.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:23.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:23.958 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:57:23.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:57:23.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:23.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:25.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:26.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:26.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:26.080 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:57:26.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:57:26.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:26.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:27.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:28.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:28.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:28.175 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:57:28.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:57:28.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:28.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:29.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:30.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:30.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:30.017 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:57:30.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:57:30.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:30.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:30.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:31.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:31.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:31.822 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:57:31.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:57:31.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:31.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:33.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:34.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:34.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:34.237 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:57:34.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:57:34.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:34.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:35.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:36.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:36.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:36.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:57:36.082 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:57:36.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:36.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:36.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:37.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:37.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:37.679 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:57:37.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:57:37.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:37.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:38.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:39.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:39.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:39.593 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:57:39.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:57:39.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:39.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:41.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:42.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:42.629 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:57:42.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:42.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:57:42.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:42.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:44.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:45.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:45.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:45.233 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:57:45.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:57:45.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:45.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:47.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:48.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:48.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:48.502 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:57:48.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:57:48.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:48.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:49.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:57:50.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:57:50.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:57:50.173 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:57:50.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:57:50.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:57:50.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T16:57:50.832 INFO:tasks.workunit.client.0.smithi037.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3305642807,v1:127.0.0.1:6835/3305642807] [v2:127.0.0.1:6836/3305642807,v1:127.0.0.1:6837/3305642807] exists,up 22a0cc88-8a3c-4ca6-9fa9-8e452e48c26d 2024-09-17T16:57:50.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 5 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/5 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:57:50.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/5' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/5/journal' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:57:50.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:57:50.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/5 2024-09-17T16:57:50.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:57:50.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=45e7f0a2-c16c-4286-8bcd-56e143a91337 2024-09-17T16:57:50.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 45e7f0a2-c16c-4286-8bcd-56e143a91337' 2024-09-17T16:57:50.842 INFO:tasks.workunit.client.0.smithi037.stdout:add osd5 45e7f0a2-c16c-4286-8bcd-56e143a91337 2024-09-17T16:57:50.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:57:50.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAOtelmRrLKMhAAhHwyGOBr0RKbz6hxLJt2UQ== 2024-09-17T16:57:50.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAOtelmRrLKMhAAhHwyGOBr0RKbz6hxLJt2UQ=="}' 2024-09-17T16:57:50.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 45e7f0a2-c16c-4286-8bcd-56e143a91337 -i td/mon-stretch-fail-recovery/5/new.json 2024-09-17T16:57:51.951 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:57:51.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/5/new.json 2024-09-17T16:57:51.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/5 --osd-journal=td/mon-stretch-fail-recovery/5/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOtelmRrLKMhAAhHwyGOBr0RKbz6hxLJt2UQ== --osd-uuid 45e7f0a2-c16c-4286-8bcd-56e143a91337 2024-09-17T16:57:51.992 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:51.989+0000 7f4ef6734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:52.032 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:52.030+0000 7f4ef6734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:52.033 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:52.031+0000 7f4ef6734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:57:52.034 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:52.031+0000 7f4ef6734640 -1 bdev(0x56228c013c00 td/mon-stretch-fail-recovery/5/block) open stat got: (1) Operation not permitted 2024-09-17T16:57:52.034 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:57:52.031+0000 7f4ef6734640 -1 bluestore(td/mon-stretch-fail-recovery/5) _read_fsid unparsable uuid 2024-09-17T16:57:59.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/5/keyring 2024-09-17T16:57:59.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:57:59.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T16:57:59.118 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd5 key to auth repository 2024-09-17T16:57:59.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:58:00.096 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.5 2024-09-17T16:58:00.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T16:58:00.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/5 --osd-journal=td/mon-stretch-fail-recovery/5/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:58:00.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:58:00.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:58:00.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:58:00.122 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:00.120+0000 7f679500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:00.124 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:00.122+0000 7f679500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:00.125 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:00.123+0000 7f679500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:00.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T16:58:00.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:58:00.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T16:58:00.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:58:00.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:58:00.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:00.718 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:58:00.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:58:00.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:00.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:01.214 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:01.212+0000 7f679500f640 -1 Falling back to public interface 2024-09-17T16:58:01.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:02.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:02.406 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:58:02.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:02.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:58:02.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:02.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:03.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:04.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:04.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:04.028 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:58:04.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:58:04.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:04.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:04.394 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:04.391+0000 7f679500f640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:58:05.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:06.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:06.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:06.033 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:58:06.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:58:06.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:06.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:07.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:08.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:08.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:08.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:58:08.502 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:58:08.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:08.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:09.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:10.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:10.488 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:10.488 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:58:10.488 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:58:10.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:10.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:11.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:12.334 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:12.334 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:12.335 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:58:12.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:58:12.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:12.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:13.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:14.235 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:14.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:14.236 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:58:14.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:58:14.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:14.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:15.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:16.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:16.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:16.222 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:58:16.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:58:16.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:16.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:17.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:18.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:18.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:18.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:58:18.397 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:58:18.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:18.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:19.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:20.482 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:58:20.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:58:20.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:20.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:21.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:22.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:22.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:22.817 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:58:22.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:58:22.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:22.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:25.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:26.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:26.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:26.272 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:58:26.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:58:26.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:26.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:30.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:31.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:31.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:31.729 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:58:31.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:58:31.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:31.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:32.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:33.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:33.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:33.811 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:58:33.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:58:33.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:33.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T16:58:34.504 INFO:tasks.workunit.client.0.smithi037.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/123624600,v1:127.0.0.1:6843/123624600] [v2:127.0.0.1:6844/123624600,v1:127.0.0.1:6845/123624600] exists,up 45e7f0a2-c16c-4286-8bcd-56e143a91337 2024-09-17T16:58:34.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:58:34.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:58:34.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 6 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/6 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:58:34.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/6' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/6/journal' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:58:34.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/6 2024-09-17T16:58:34.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:58:34.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=44806de4-95d2-40af-8787-32d08892193a 2024-09-17T16:58:34.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 44806de4-95d2-40af-8787-32d08892193a' 2024-09-17T16:58:34.512 INFO:tasks.workunit.client.0.smithi037.stdout:add osd6 44806de4-95d2-40af-8787-32d08892193a 2024-09-17T16:58:34.512 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:58:34.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6telmKhVHHxAAP/3L+gG4LSSPYxeC4fb8qw== 2024-09-17T16:58:34.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6telmKhVHHxAAP/3L+gG4LSSPYxeC4fb8qw=="}' 2024-09-17T16:58:34.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 44806de4-95d2-40af-8787-32d08892193a -i td/mon-stretch-fail-recovery/6/new.json 2024-09-17T16:58:36.317 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:58:36.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/6/new.json 2024-09-17T16:58:36.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/6 --osd-journal=td/mon-stretch-fail-recovery/6/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6telmKhVHHxAAP/3L+gG4LSSPYxeC4fb8qw== --osd-uuid 44806de4-95d2-40af-8787-32d08892193a 2024-09-17T16:58:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:36.355+0000 7fedb5e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:36.414 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:36.411+0000 7fedb5e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:36.415 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:36.412+0000 7fedb5e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:36.415 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:36.413+0000 7fedb5e22640 -1 bdev(0x556fec13bc00 td/mon-stretch-fail-recovery/6/block) open stat got: (1) Operation not permitted 2024-09-17T16:58:36.415 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:36.413+0000 7fedb5e22640 -1 bluestore(td/mon-stretch-fail-recovery/6) _read_fsid unparsable uuid 2024-09-17T16:58:42.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/6/keyring 2024-09-17T16:58:42.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:58:42.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T16:58:42.982 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd6 key to auth repository 2024-09-17T16:58:42.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:58:43.557 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.6 2024-09-17T16:58:43.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T16:58:43.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/6 --osd-journal=td/mon-stretch-fail-recovery/6/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:58:43.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:58:43.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:58:43.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:58:43.583 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:43.581+0000 7f32dd431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:43.604 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:43.602+0000 7f32dd431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:43.605 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:43.603+0000 7f32dd431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:58:44.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T16:58:44.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:58:44.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T16:58:44.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:58:44.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:58:44.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:44.114 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:58:44.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:58:44.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:44.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:44.720 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:44.718+0000 7f32dd431640 -1 Falling back to public interface 2024-09-17T16:58:44.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:45.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:45.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:45.779 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:58:45.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:58:45.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:45.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:46.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:47.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:47.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:47.396 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:58:47.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:58:47.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:47.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:47.457 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:58:47.456+0000 7f32dd431640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:58:48.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:49.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:49.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:49.413 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:58:49.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:58:49.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:49.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:50.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:58:51.635 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:58:51.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:51.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:52.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:53.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:53.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:53.646 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:58:53.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:58:53.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:53.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:54.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:55.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:55.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:55.516 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:58:55.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:58:55.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:55.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:56.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:57.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:57.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:57.477 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:58:57.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:58:57.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:57.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:58:58.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:58:59.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:58:59.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:58:59.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:58:59.059 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:58:59.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:58:59.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:00.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:01.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:01.081 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:59:01.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:01.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:59:01.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:01.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:02.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:03.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:03.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:03.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:59:03.112 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:59:03.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:03.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:04.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:05.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:05.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:05.193 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:59:05.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:59:05.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:05.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:06.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:07.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:07.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:07.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:59:07.381 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:59:07.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:07.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:09.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:10.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:10.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:10.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:59:10.218 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:59:10.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:10.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:15.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:16.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:16.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:16.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:59:16.141 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:59:16.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:16.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:16.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:17.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:17.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:17.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T16:59:17.826 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T16:59:17.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:17.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T16:59:18.477 INFO:tasks.workunit.client.0.smithi037.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2140153412,v1:127.0.0.1:6851/2140153412] [v2:127.0.0.1:6852/2140153412,v1:127.0.0.1:6853/2140153412] exists,up 44806de4-95d2-40af-8787-32d08892193a 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 7 2024-09-17T16:59:18.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/7 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/7' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/7/journal' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T16:59:18.482 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T16:59:18.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/7 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c6153248-8aeb-4f40-bd86-4b9877f9f6e0 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 c6153248-8aeb-4f40-bd86-4b9877f9f6e0' 2024-09-17T16:59:18.484 INFO:tasks.workunit.client.0.smithi037.stdout:add osd7 c6153248-8aeb-4f40-bd86-4b9877f9f6e0 2024-09-17T16:59:18.485 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T16:59:18.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBmtelmo9KdHRAAvvmy3U2LOO+4BD/8AZWnUQ== 2024-09-17T16:59:18.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBmtelmo9KdHRAAvvmy3U2LOO+4BD/8AZWnUQ=="}' 2024-09-17T16:59:18.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c6153248-8aeb-4f40-bd86-4b9877f9f6e0 -i td/mon-stretch-fail-recovery/7/new.json 2024-09-17T16:59:19.392 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:59:19.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/7/new.json 2024-09-17T16:59:19.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/7 --osd-journal=td/mon-stretch-fail-recovery/7/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmtelmo9KdHRAAvvmy3U2LOO+4BD/8AZWnUQ== --osd-uuid c6153248-8aeb-4f40-bd86-4b9877f9f6e0 2024-09-17T16:59:19.433 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:19.430+0000 7f0c8e620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:19.435 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:19.432+0000 7f0c8e620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:19.436 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:19.433+0000 7f0c8e620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:19.436 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:19.434+0000 7f0c8e620640 -1 bdev(0x557814e89c00 td/mon-stretch-fail-recovery/7/block) open stat got: (1) Operation not permitted 2024-09-17T16:59:19.436 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:19.434+0000 7f0c8e620640 -1 bluestore(td/mon-stretch-fail-recovery/7) _read_fsid unparsable uuid 2024-09-17T16:59:25.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/7/keyring 2024-09-17T16:59:25.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T16:59:25.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-17T16:59:25.803 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd7 key to auth repository 2024-09-17T16:59:25.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T16:59:26.532 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.7 2024-09-17T16:59:26.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-17T16:59:26.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/7 --osd-journal=td/mon-stretch-fail-recovery/7/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T16:59:26.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T16:59:26.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T16:59:26.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T16:59:26.558 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:26.556+0000 7f4dfd60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:26.587 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:26.584+0000 7f4dfd60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:26.588 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:26.586+0000 7f4dfd60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T16:59:27.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-17T16:59:27.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T16:59:27.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-17T16:59:27.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T16:59:27.224 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T16:59:27.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T16:59:27.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:27.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T16:59:27.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:27.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:27.696 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:27.694+0000 7f4dfd60b640 -1 Falling back to public interface 2024-09-17T16:59:30.380 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T16:59:30.377+0000 7f4dfd60b640 -1 osd.7 0 log_to_monitors true 2024-09-17T16:59:30.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:31.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:31.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:31.994 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T16:59:31.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T16:59:31.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:31.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:33.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:34.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:34.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:34.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T16:59:34.329 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T16:59:34.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:34.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:34.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:35.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:35.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:35.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T16:59:35.952 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T16:59:35.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:35.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:36.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:37.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:37.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:37.893 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T16:59:37.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T16:59:37.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:37.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:39.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:40.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:40.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:40.056 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T16:59:40.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T16:59:40.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:40.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:41.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:42.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:42.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:42.264 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T16:59:42.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T16:59:42.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:42.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:42.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:43.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:43.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:43.961 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T16:59:43.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T16:59:43.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:43.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:44.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:45.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:45.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:45.434 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T16:59:45.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T16:59:45.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:45.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:46.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:47.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:47.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:47.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T16:59:47.038 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T16:59:47.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:47.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:48.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:49.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:49.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:49.338 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T16:59:49.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T16:59:49.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:49.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:51.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:52.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:52.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:52.139 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T16:59:52.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T16:59:52.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:52.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:53.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:54.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:54.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:54.246 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T16:59:54.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T16:59:54.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:54.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:56.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:57.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:57.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:57.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T16:59:57.331 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T16:59:57.331 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:57.331 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T16:59:58.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T16:59:59.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T16:59:59.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T16:59:59.707 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T16:59:59.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T16:59:59.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T16:59:59.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T17:00:03.278 INFO:tasks.workunit.client.0.smithi037.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4066917517,v1:127.0.0.1:6859/4066917517] [v2:127.0.0.1:6860/4066917517,v1:127.0.0.1:6861/4066917517] exists,up c6153248-8aeb-4f40-bd86-4b9877f9f6e0 2024-09-17T17:00:03.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:00:03.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:00:03.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:00:03.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:62: TEST_stretched_cluster_failover_add_three_osds: for zone in iris pze 2024-09-17T17:00:03.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:64: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket iris zone 2024-09-17T17:00:04.687 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'iris' already exists 2024-09-17T17:00:04.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:65: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move iris root=default 2024-09-17T17:00:05.654 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -5 name 'iris' to location {root=default} in crush map 2024-09-17T17:00:05.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:62: TEST_stretched_cluster_failover_add_three_osds: for zone in iris pze 2024-09-17T17:00:05.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:64: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket pze zone 2024-09-17T17:00:07.096 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'pze' already exists 2024-09-17T17:00:07.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:65: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move pze root=default 2024-09-17T17:00:08.325 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -7 name 'pze' to location {root=default} in crush map 2024-09-17T17:00:08.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:69: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-2 host 2024-09-17T17:00:09.700 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-2' already exists 2024-09-17T17:00:09.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:70: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-3 host 2024-09-17T17:00:11.281 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-3' already exists 2024-09-17T17:00:11.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:71: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-4 host 2024-09-17T17:00:12.136 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-4' already exists 2024-09-17T17:00:12.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:72: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-5 host 2024-09-17T17:00:16.389 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-5' already exists 2024-09-17T17:00:16.401 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:74: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-2 zone=iris 2024-09-17T17:00:17.570 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -9 name 'node-2' to location {zone=iris} in crush map 2024-09-17T17:00:17.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:75: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-3 zone=iris 2024-09-17T17:00:18.860 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -10 name 'node-3' to location {zone=iris} in crush map 2024-09-17T17:00:18.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:76: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-4 zone=pze 2024-09-17T17:00:20.075 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -11 name 'node-4' to location {zone=pze} in crush map 2024-09-17T17:00:20.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:77: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-5 zone=pze 2024-09-17T17:00:21.040 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -12 name 'node-5' to location {zone=pze} in crush map 2024-09-17T17:00:21.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:79: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.0 host=node-2 2024-09-17T17:00:22.296 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 0 name 'osd.0' to location {host=node-2} in crush map 2024-09-17T17:00:22.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:80: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.1 host=node-2 2024-09-17T17:00:23.230 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 1 name 'osd.1' to location {host=node-2} in crush map 2024-09-17T17:00:23.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:81: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.2 host=node-3 2024-09-17T17:00:23.831 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 2 name 'osd.2' to location {host=node-3} in crush map 2024-09-17T17:00:23.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:82: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.3 host=node-3 2024-09-17T17:00:24.979 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 3 name 'osd.3' to location {host=node-3} in crush map 2024-09-17T17:00:24.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:83: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.4 host=node-4 2024-09-17T17:00:26.037 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 4 name 'osd.4' to location {host=node-4} in crush map 2024-09-17T17:00:26.050 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:84: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.5 host=node-4 2024-09-17T17:00:26.955 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 5 name 'osd.5' to location {host=node-4} in crush map 2024-09-17T17:00:26.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:85: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.6 host=node-5 2024-09-17T17:00:27.855 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 6 name 'osd.6' to location {host=node-5} in crush map 2024-09-17T17:00:27.867 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:86: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.7 host=node-5 2024-09-17T17:00:28.405 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 7 name 'osd.7' to location {host=node-5} in crush map 2024-09-17T17:00:28.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:88: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location a zone=iris host=node-2 2024-09-17T17:00:32.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:89: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location b zone=iris host=node-3 2024-09-17T17:00:33.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:90: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location c zone=pze host=node-4 2024-09-17T17:00:41.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:91: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location d zone=pze host=node-5 2024-09-17T17:00:53.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:93: TEST_stretched_cluster_failover_add_three_osds: hostname -s 2024-09-17T17:00:53.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:93: TEST_stretched_cluster_failover_add_three_osds: hostname=smithi037 2024-09-17T17:00:53.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:94: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush remove smithi037 2024-09-17T17:00:55.106 INFO:tasks.workunit.client.0.smithi037.stderr:device 'smithi037' does not appear in the crush map 2024-09-17T17:00:55.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:95: TEST_stretched_cluster_failover_add_three_osds: ceph osd getcrushmap 2024-09-17T17:00:55.573 INFO:tasks.workunit.client.0.smithi037.stderr:38 2024-09-17T17:00:55.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:96: TEST_stretched_cluster_failover_add_three_osds: crushtool --decompile crushmap 2024-09-17T17:00:55.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:97: TEST_stretched_cluster_failover_add_three_osds: sed 's/^# end crush map$//' crushmap.txt 2024-09-17T17:00:55.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:98: TEST_stretched_cluster_failover_add_three_osds: cat 2024-09-17T17:00:55.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:115: TEST_stretched_cluster_failover_add_three_osds: crushtool --compile crushmap_modified.txt -o crushmap.bin 2024-09-17T17:00:55.619 INFO:tasks.workunit.client.0.smithi037.stderr:WARNING: min_size is no longer supported, ignoring 2024-09-17T17:00:55.619 INFO:tasks.workunit.client.0.smithi037.stderr:WARNING: max_size is no longer supported, ignoring 2024-09-17T17:00:55.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:116: TEST_stretched_cluster_failover_add_three_osds: ceph osd setcrushmap -i crushmap.bin 2024-09-17T17:00:57.325 INFO:tasks.workunit.client.0.smithi037.stderr:40 2024-09-17T17:00:57.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:117: TEST_stretched_cluster_failover_add_three_osds: local stretched_poolname=stretched_rbdpool 2024-09-17T17:00:57.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:118: TEST_stretched_cluster_failover_add_three_osds: ceph osd pool create stretched_rbdpool 32 32 stretch_rule 2024-09-17T17:00:58.121 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'stretched_rbdpool' already exists 2024-09-17T17:00:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:119: TEST_stretched_cluster_failover_add_three_osds: ceph osd pool set stretched_rbdpool size 4 2024-09-17T17:01:00.462 INFO:tasks.workunit.client.0.smithi037.stderr:set pool 1 size to 4 2024-09-17T17:01:00.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:121: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2024-09-17T17:01:03.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:123: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location e zone=arbiter host=node-1 2024-09-17T17:01:05.331 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:124: TEST_stretched_cluster_failover_add_three_osds: ceph mon enable_stretch_mode e stretch_rule zone 2024-09-17T17:01:07.122 INFO:tasks.workunit.client.0.smithi037.stderr:Second attempt of previously successful command failed with EINVAL: stretch mode is already engaged 2024-09-17T17:01:07.122 INFO:tasks.workunit.client.0.smithi037.stderr:stretch mode is already engaged 2024-09-17T17:01:07.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:126: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL mon.c 2024-09-17T17:01:07.134 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.134 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:127: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL mon.d 2024-09-17T17:01:07.240 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:129: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.4 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:130: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.5 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:131: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.6 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:132: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.7 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:01:07.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:01:07.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:01:07.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:134: TEST_stretched_cluster_failover_add_three_osds: ceph -s 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: cluster: 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: id: bcd61817-a54f-4023-86d3-fc20ff4fbf27 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: health: HEALTH_WARN 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: 1 pool(s) do not have an application enabled 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: services: 2024-09-17T17:01:08.307 INFO:tasks.workunit.client.0.smithi037.stdout: mon: 5 daemons, quorum a,b,c,d,e (age 1.32063s) 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: mgr: x(active, since 6m), standbys: y, z 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: osd: 8 osds: 8 up (since 68s), 8 in (since 109s) 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: data: 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: pools: 1 pools, 32 pgs 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: objects: 0 objects, 0 B 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: usage: 215 MiB used, 800 GiB / 800 GiB avail 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: pgs: 37.500% pgs not active 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: 20 active+clean 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: 7 activating 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: 5 creating+activating 2024-09-17T17:01:08.308 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:01:08.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:136: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2024-09-17T17:01:11.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:138: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 8 2024-09-17T17:01:11.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:01:11.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:01:11.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/8 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/8' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/8/journal' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:01:11.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T17:01:11.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/8 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:01:11.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e44fd53-6a41-4d89-9a65-38602d9186ea 2024-09-17T17:01:11.327 INFO:tasks.workunit.client.0.smithi037.stdout:add osd8 7e44fd53-6a41-4d89-9a65-38602d9186ea 2024-09-17T17:01:11.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 7e44fd53-6a41-4d89-9a65-38602d9186ea' 2024-09-17T17:01:11.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:01:11.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDXtelm0GqBFBAA+lGowbbdEAjGjklAnktuQA== 2024-09-17T17:01:11.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDXtelm0GqBFBAA+lGowbbdEAjGjklAnktuQA=="}' 2024-09-17T17:01:11.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e44fd53-6a41-4d89-9a65-38602d9186ea -i td/mon-stretch-fail-recovery/8/new.json 2024-09-17T17:01:34.837 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:01:34.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/8/new.json 2024-09-17T17:01:34.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/8 --osd-journal=td/mon-stretch-fail-recovery/8/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXtelm0GqBFBAA+lGowbbdEAjGjklAnktuQA== --osd-uuid 7e44fd53-6a41-4d89-9a65-38602d9186ea 2024-09-17T17:01:34.884 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:34.883+0000 7f7fd080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:34.922 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:34.920+0000 7f7fd080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:34.923 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:34.922+0000 7f7fd080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:34.924 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:34.922+0000 7f7fd080d640 -1 bdev(0x55e425fefc00 td/mon-stretch-fail-recovery/8/block) open stat got: (1) Operation not permitted 2024-09-17T17:01:34.924 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:34.923+0000 7f7fd080d640 -1 bluestore(td/mon-stretch-fail-recovery/8) _read_fsid unparsable uuid 2024-09-17T17:01:41.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/8/keyring 2024-09-17T17:01:41.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:01:41.079 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd8 key to auth repository 2024-09-17T17:01:41.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-17T17:01:41.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:01:41.763 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.8 2024-09-17T17:01:41.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-17T17:01:41.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/8 --osd-journal=td/mon-stretch-fail-recovery/8/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:01:41.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:01:41.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:01:41.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:01:41.789 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:41.787+0000 7fa75b8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:44.793 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:44.790+0000 7fa75b8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:44.794 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:44.792+0000 7fa75b8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:01:45.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-17T17:01:45.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:01:45.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-17T17:01:45.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:01:45.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:01:45.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:45.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:01:45.314 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:01:45.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:45.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:01:45.648 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:45.646+0000 7fa75b8e2640 -1 Falling back to public interface 2024-09-17T17:01:45.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:46.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:46.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:46.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:01:46.994 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:01:46.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:46.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:01:48.221 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:01:48.218+0000 7fa75b8e2640 -1 osd.8 0 log_to_monitors true 2024-09-17T17:01:50.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:51.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:51.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:51.644 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:01:51.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:01:51.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:51.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:01:52.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:53.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:53.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:53.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:01:53.356 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:01:53.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:53.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:01:53.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:54.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:54.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:54.792 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:01:54.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:01:54.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:54.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:01:58.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:01:59.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:01:59.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:01:59.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:01:59.470 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:01:59.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:01:59.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:00.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:01.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:01.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:01.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:02:01.043 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:02:01.044 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:01.044 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:01.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:02:02.922 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:02:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:02.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:04.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:05.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:05.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:05.450 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:02:05.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:02:05.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:05.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:09.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:10.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:10.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:10.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:02:10.270 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:02:10.271 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:10.271 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:12.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:13.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:13.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:13.292 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:02:13.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:02:13.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:13.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T17:02:16.834 INFO:tasks.workunit.client.0.smithi037.stdout:osd.8 up in weight 1 up_from 82 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1890117020,v1:127.0.0.1:6835/1890117020] [v2:127.0.0.1:6836/1890117020,v1:127.0.0.1:6837/1890117020] exists,up 7e44fd53-6a41-4d89-9a65-38602d9186ea 2024-09-17T17:02:16.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:02:16.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:02:16.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:02:16.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:139: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 9 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/9 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/9' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/9/journal' 2024-09-17T17:02:16.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:02:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/9 2024-09-17T17:02:16.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:02:16.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=afd324b3-73ec-4594-896c-0032a8b0e129 2024-09-17T17:02:16.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 afd324b3-73ec-4594-896c-0032a8b0e129' 2024-09-17T17:02:16.842 INFO:tasks.workunit.client.0.smithi037.stdout:add osd9 afd324b3-73ec-4594-896c-0032a8b0e129 2024-09-17T17:02:16.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:02:16.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAYtulmBX7kMhAAkRyuhewBPz501IChshqwAg== 2024-09-17T17:02:16.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAYtulmBX7kMhAAkRyuhewBPz501IChshqwAg=="}' 2024-09-17T17:02:16.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new afd324b3-73ec-4594-896c-0032a8b0e129 -i td/mon-stretch-fail-recovery/9/new.json 2024-09-17T17:02:20.369 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:02:20.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/9/new.json 2024-09-17T17:02:20.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/9 --osd-journal=td/mon-stretch-fail-recovery/9/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYtulmBX7kMhAAkRyuhewBPz501IChshqwAg== --osd-uuid afd324b3-73ec-4594-896c-0032a8b0e129 2024-09-17T17:02:20.409 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:20.407+0000 7f8c84f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:23.413 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:23.411+0000 7f8c84f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:23.414 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:23.412+0000 7f8c84f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:23.414 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:23.413+0000 7f8c84f45640 -1 bdev(0x55ff70080800 td/mon-stretch-fail-recovery/9/block) open stat got: (1) Operation not permitted 2024-09-17T17:02:23.414 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:23.413+0000 7f8c84f45640 -1 bluestore(td/mon-stretch-fail-recovery/9) _read_fsid unparsable uuid 2024-09-17T17:02:29.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/9/keyring 2024-09-17T17:02:29.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:02:29.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-17T17:02:29.365 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd9 key to auth repository 2024-09-17T17:02:29.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:02:29.971 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.9 2024-09-17T17:02:29.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-17T17:02:29.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/9 --osd-journal=td/mon-stretch-fail-recovery/9/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:02:29.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:02:29.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:02:29.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:02:29.998 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:29.997+0000 7f552a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:30.075 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:30.074+0000 7f552a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:30.076 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:30.075+0000 7f552a20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:02:30.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-17T17:02:30.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:02:30.375 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:02:30.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:30.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:31.407 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:31.406+0000 7f552a20c640 -1 Falling back to public interface 2024-09-17T17:02:34.086 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:02:34.085+0000 7f552a20c640 -1 osd.9 0 log_to_monitors true 2024-09-17T17:02:36.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:37.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:37.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:37.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:02:37.946 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:02:37.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:37.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:41.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:42.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:42.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:42.472 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:02:42.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:02:42.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:42.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:43.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:44.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:44.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:44.241 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:02:44.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:02:44.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:44.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:51.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:52.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:52.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:52.030 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:02:52.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:02:52.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:52.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:54.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:55.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:55.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:55.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:02:55.059 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:02:55.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:55.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:56.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:02:57.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:02:57.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:02:57.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:02:57.797 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:02:57.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:02:57.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:02:59.454 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:03:00.456 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:03:00.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:00.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:03:01.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:02.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:02.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:02.392 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:03:02.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:03:02.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:02.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T17:03:03.010 INFO:tasks.workunit.client.0.smithi037.stdout:osd.9 up in weight 1 up_from 87 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2879566434,v1:127.0.0.1:6843/2879566434] [v2:127.0.0.1:6844/2879566434,v1:127.0.0.1:6845/2879566434] exists,up afd324b3-73ec-4594-896c-0032a8b0e129 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:140: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 10 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:03:03.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/10 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/10' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/10/journal' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2024-09-17T17:03:03.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2024-09-17T17:03:03.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/10 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=391b40f7-c1b8-4c26-afb4-16305146bd0c 2024-09-17T17:03:03.018 INFO:tasks.workunit.client.0.smithi037.stdout:add osd10 391b40f7-c1b8-4c26-afb4-16305146bd0c 2024-09-17T17:03:03.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 391b40f7-c1b8-4c26-afb4-16305146bd0c' 2024-09-17T17:03:03.019 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:03:03.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHtulmJY3EARAALzDV2VkQfJlefhMhk+z7Hg== 2024-09-17T17:03:03.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHtulmJY3EARAALzDV2VkQfJlefhMhk+z7Hg=="}' 2024-09-17T17:03:03.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 391b40f7-c1b8-4c26-afb4-16305146bd0c -i td/mon-stretch-fail-recovery/10/new.json 2024-09-17T17:03:06.540 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:03:06.552 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/10/new.json 2024-09-17T17:03:06.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/10 --osd-journal=td/mon-stretch-fail-recovery/10/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHtulmJY3EARAALzDV2VkQfJlefhMhk+z7Hg== --osd-uuid 391b40f7-c1b8-4c26-afb4-16305146bd0c 2024-09-17T17:03:06.581 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:06.578+0000 7fbf2f00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:06.598 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:06.596+0000 7fbf2f00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:06.599 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:06.597+0000 7fbf2f00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:06.600 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:06.597+0000 7fbf2f00d640 -1 bdev(0x55d589885c00 td/mon-stretch-fail-recovery/10/block) open stat got: (1) Operation not permitted 2024-09-17T17:03:06.600 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:06.598+0000 7fbf2f00d640 -1 bluestore(td/mon-stretch-fail-recovery/10) _read_fsid unparsable uuid 2024-09-17T17:03:13.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/10/keyring 2024-09-17T17:03:13.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:03:13.286 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd10 key to auth repository 2024-09-17T17:03:13.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2024-09-17T17:03:13.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:03:13.932 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.10 2024-09-17T17:03:13.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2024-09-17T17:03:13.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=bcd61817-a54f-4023-86d3-fc20ff4fbf27 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/10 --osd-journal=td/mon-stretch-fail-recovery/10/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.27368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:03:13.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:03:13.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:03:13.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:03:13.957 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:13.955+0000 7f94d3839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:13.959 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:13.957+0000 7f94d3839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:13.960 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:13.958+0000 7f94d3839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:03:15.069 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:15.066+0000 7f94d3839640 -1 Falling back to public interface 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:17.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:03:17.578 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:03:17.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:17.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:17.690 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:03:17.688+0000 7f94d3839640 -1 osd.10 0 log_to_monitors true 2024-09-17T17:03:18.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:19.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:19.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:19.315 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:03:19.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:03:19.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:19.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:23.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:24.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:24.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:24.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:03:24.221 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:03:24.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:24.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:25.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:26.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:26.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:26.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:03:26.026 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:03:26.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:26.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:26.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:27.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:27.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:27.474 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:03:27.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:03:27.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:27.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:28.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:29.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:29.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:29.104 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:03:29.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:03:29.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:29.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:30.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:31.160 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:31.160 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:31.160 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:03:31.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:03:31.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:31.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:31.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:32.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:32.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:32.983 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:03:32.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:03:32.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:32.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:33.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:34.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:34.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:34.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:03:34.827 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:03:34.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:34.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:42.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:43.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:43.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:43.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:03:43.992 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:03:43.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:43.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:45.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:46.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:46.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:46.201 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:03:46.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:03:46.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:46.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:47.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:03:48.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:03:48.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:03:48.333 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T17:03:48.334 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:03:48.334 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:03:48.334 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-17T17:03:48.762 INFO:tasks.workunit.client.0.smithi037.stdout:osd.10 up in weight 1 up_from 92 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/505775823,v1:127.0.0.1:6851/505775823] [v2:127.0.0.1:6852/505775823,v1:127.0.0.1:6853/505775823] exists,up 391b40f7-c1b8-4c26-afb4-16305146bd0c 2024-09-17T17:03:48.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:03:48.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:03:48.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:03:48.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:142: TEST_stretched_cluster_failover_add_three_osds: ceph -s 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: cluster: 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: id: bcd61817-a54f-4023-86d3-fc20ff4fbf27 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: health: HEALTH_WARN 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: We are missing stretch mode buckets, only requiring 1 of 2 buckets to peer 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: 2/5 mons down, quorum a,b,e 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: 4 osds down 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: 2 hosts (4 osds) down 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: 1 zone (4 osds) down 2024-09-17T17:03:52.379 INFO:tasks.workunit.client.0.smithi037.stdout: Degraded data redundancy: 32 pgs undersized 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: 1 pool(s) do not have an application enabled 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: services: 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: mon: 5 daemons, quorum a,b,e (age 2m), out of quorum: c, d 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: mgr: x(active, since 9m), standbys: y, z 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: osd: 11 osds: 7 up (since 4s), 11 in (since 46s) 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: data: 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: pools: 1 pools, 32 pgs 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: objects: 0 objects, 0 B 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: usage: 698 MiB used, 1.1 TiB / 1.1 TiB avail 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: pgs: 32 active+undersized 2024-09-17T17:03:52.380 INFO:tasks.workunit.client.0.smithi037.stdout: 2024-09-17T17:03:52.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:144: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2024-09-17T17:03:55.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:146: TEST_stretched_cluster_failover_add_three_osds: teardown td/mon-stretch-fail-recovery 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:03:55.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:03:55.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:03:55.519 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:03:55.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:03:55.520 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:03:55.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:03:55.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:03:55.522 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:03:55.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:03:55.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:03:55.523 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:03:55.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:03:55.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2024-09-17T17:03:55.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:03:55.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:03:55.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27368 2024-09-17T17:03:55.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:03:55.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:03:55.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:23: run: teardown td/mon-stretch-fail-recovery 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:03:55.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:03:55.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:03:55.677 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:03:55.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:03:55.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:03:55.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:03:55.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:03:55.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:03:55.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:03:55.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:03:55.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.682 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:03:55.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:03:55.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2024-09-17T17:03:55.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:03:55.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:03:55.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27368 2024-09-17T17:03:55.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:03:55.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/mon-stretch-fail-recovery 0 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:03:55.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:03:55.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:03:55.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:03:55.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:03:55.689 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:03:55.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:03:55.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:03:55.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:03:55.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:03:55.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:03:55.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:03:55.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T17:03:55.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2024-09-17T17:03:55.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:03:55.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.695 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27368 2024-09-17T17:03:55.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27368 2024-09-17T17:03:55.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:03:55.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:03:55.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T17:03:55.697 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T17:03:55.697 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T17:03:55.726 INFO:tasks.workunit:Running workunit mon-stretch/mon-stretch-uneven-crush-weights.sh... 2024-09-17T17:03:55.726 DEBUG:teuthology.orchestra.run.smithi037:workunit test mon-stretch/mon-stretch-uneven-crush-weights.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh 2024-09-17T17:03:55.788 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T17:03:55.794 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T17:03:55.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:03:55.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:5: run: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:6: run: shift 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:8: run: export CEPH_MON_A=127.0.0.1:7139 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:8: run: CEPH_MON_A=127.0.0.1:7139 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:9: run: export CEPH_MON_B=127.0.0.1:7141 2024-09-17T17:03:55.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:9: run: CEPH_MON_B=127.0.0.1:7141 2024-09-17T17:03:55.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:10: run: export CEPH_MON_C=127.0.0.1:7142 2024-09-17T17:03:55.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:10: run: CEPH_MON_C=127.0.0.1:7142 2024-09-17T17:03:55.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:11: run: export CEPH_MON_D=127.0.0.1:7143 2024-09-17T17:03:55.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:11: run: CEPH_MON_D=127.0.0.1:7143 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:12: run: export CEPH_MON_E=127.0.0.1:7144 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:12: run: CEPH_MON_E=127.0.0.1:7144 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:13: run: export CEPH_ARGS 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:14: run: uuidgen 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:14: run: CEPH_ARGS+='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none ' 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:16: run: export 'BASE_CEPH_ARGS=--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none ' 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:16: run: BASE_CEPH_ARGS='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none ' 2024-09-17T17:03:55.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:17: run: CEPH_ARGS+=--mon-host=127.0.0.1:7139 2024-09-17T17:03:55.798 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: set 2024-09-17T17:03:55.798 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T17:03:55.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: local funcs=TEST_stretched_cluster_uneven_weight 2024-09-17T17:03:55.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:20: run: for func in $funcs 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:21: run: setup td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:03:55.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:03:55.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:03:55.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:03:55.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:03:55.802 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:03:55.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:03:55.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:03:55.803 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:03:55.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:03:55.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.804 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:03:55.804 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.804 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:55.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47588 2024-09-17T17:03:55.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:03:55.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:03:55.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.807 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:03:55.807 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.807 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:55.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.47588 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretched-cluster-uneven-weight 1' TERM HUP INT 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:22: run: TEST_stretched_cluster_uneven_weight td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:27: TEST_stretched_cluster_uneven_weight: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:28: TEST_stretched_cluster_uneven_weight: local OSDS=4 2024-09-17T17:03:55.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:29: TEST_stretched_cluster_uneven_weight: local weight=0.09000 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:30: TEST_stretched_cluster_uneven_weight: setup td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:03:55.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:03:55.813 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:03:55.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:03:55.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:03:55.814 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:03:55.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:03:55.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:03:55.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47588 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:03:55.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:03:55.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:55.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:55.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.47588 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretched-cluster-uneven-weight 1' TERM HUP INT 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:32: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight a --public-addr 127.0.0.1:7139 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/a 2024-09-17T17:03:55.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/a --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7139 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:03:56.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/a '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7139 2024-09-17T17:03:56.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:03:56.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:03:56.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:03:56.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:03:56.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:03:56.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:03:56.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:03:56.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:03:56.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:03:56.447 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:03:56.448 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:56.448 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:56.448 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.a.asok 2024-09-17T17:03:56.448 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:03:56.448 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.a.asok config get fsid 2024-09-17T17:03:56.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:03:56.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:03:56.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:03:56.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:03:56.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.a.asok 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:03:56.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.a.asok config get mon_host 2024-09-17T17:03:56.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:33: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 1 2024-09-17T17:03:56.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T17:03:56.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=1 2024-09-17T17:03:56.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T17:03:56.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 1 ]] 2024-09-17T17:03:56.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T17:03:56.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T17:03:56.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592936 2024-09-17T17:03:56.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T17:03:56.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592636 -lt 1726592936 ]] 2024-09-17T17:03:56.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 1' 2024-09-17T17:03:56.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T17:03:57.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T17:03:57.414 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:03:55.840004Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:03:57.414 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:03:55.840004Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:03:57.414 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 1' 2024-09-17T17:03:57.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:35: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight b --public-addr 127.0.0.1:7141 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/b 2024-09-17T17:03:57.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/b --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7141 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:57.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:57.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:03:57.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/b '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7141 2024-09-17T17:03:58.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:03:58.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-17T17:03:58.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:03:58.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-17T17:03:58.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:03:58.129 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:03:58.129 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.b.asok 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:03:58.130 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.b.asok config get fsid 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-17T17:03:58.218 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:03:58.219 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:03:58.219 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:03:58.219 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.b.asok 2024-09-17T17:03:58.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:03:58.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.b.asok config get mon_host 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:36: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141' 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:37: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 2 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T17:03:58.300 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T17:03:58.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592938 2024-09-17T17:03:58.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T17:03:58.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592638 -lt 1726592938 ]] 2024-09-17T17:03:58.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-17T17:03:58.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T17:04:04.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T17:04:04.505 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":2,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:03:58.126757Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:04.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":2,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:03:58.126757Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:04.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T17:04:04.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:39: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight c --public-addr 127.0.0.1:7142 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/c 2024-09-17T17:04:04.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/c --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7142 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:05.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:05.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:05.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/c '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7142 2024-09-17T17:04:05.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:04:05.287 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-17T17:04:05.288 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:05.288 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-17T17:04:05.288 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:04:05.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:04:05.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-17T17:04:05.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-17T17:04:05.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-17T17:04:05.289 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:05.290 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:05.290 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:05.290 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.c.asok 2024-09-17T17:04:05.290 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:05.290 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.c.asok config get fsid 2024-09-17T17:04:05.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-17T17:04:05.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.c.asok 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:05.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.c.asok config get mon_host 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:40: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142' 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:41: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 3 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T17:04:05.464 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T17:04:05.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592945 2024-09-17T17:04:05.465 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T17:04:05.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592645 -lt 1726592945 ]] 2024-09-17T17:04:05.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-17T17:04:05.466 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T17:04:14.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T17:04:14.671 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":3,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:05.448778Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:14.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":3,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:05.448778Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:14.671 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:43: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight d --public-addr 127.0.0.1:7143 2024-09-17T17:04:14.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:14.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:04:14.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-17T17:04:14.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:04:14.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/d 2024-09-17T17:04:14.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/d --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7143 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:15.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/d '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7143 2024-09-17T17:04:15.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:04:15.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-17T17:04:15.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:15.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-17T17:04:15.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:04:15.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:04:15.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-17T17:04:15.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-17T17:04:15.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.d.asok 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:15.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.d.asok config get fsid 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:15.552 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:15.553 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:15.553 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.d.asok 2024-09-17T17:04:15.553 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:15.553 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.d.asok config get mon_host 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:44: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143' 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:45: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 4 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T17:04:15.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T17:04:15.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592955 2024-09-17T17:04:15.637 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T17:04:15.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592655 -lt 1726592955 ]] 2024-09-17T17:04:15.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-17T17:04:15.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T17:04:21.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T17:04:21.016 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":4,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:15.601642Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:21.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":4,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:15.601642Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:21.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:47: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight e --public-addr 127.0.0.1:7144 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2024-09-17T17:04:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:04:21.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/e 2024-09-17T17:04:21.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/e --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7144 2024-09-17T17:04:21.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:04:21.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:21.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:21.982 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:21.983 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:21.983 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:21.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:21.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/e '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7144 2024-09-17T17:04:22.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:04:22.162 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-17T17:04:22.163 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-17T17:04:22.163 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-17T17:04:22.163 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:22.163 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:22.163 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:22.164 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.e.asok 2024-09-17T17:04:22.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:22.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.e.asok config get fsid 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:04:22.253 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:22.254 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:22.254 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47588/ceph-mon.e.asok 2024-09-17T17:04:22.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:04:22.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47588/ceph-mon.e.asok config get mon_host 2024-09-17T17:04:22.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:48: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:04:22.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:49: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 5 2024-09-17T17:04:22.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2024-09-17T17:04:22.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2024-09-17T17:04:22.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2024-09-17T17:04:22.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2024-09-17T17:04:22.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-17T17:04:22.338 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-17T17:04:22.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1726592962 2024-09-17T17:04:22.339 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-17T17:04:22.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1726592662 -lt 1726592962 ]] 2024-09-17T17:04:22.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-17T17:04:22.339 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2024-09-17T17:04:28.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-17T17:04:28.800 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":5,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:22.304355Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:28.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":5,"fsid":"bbedb450-ae41-45e6-a933-4e20c134f4c7","modified":"2024-09-17T17:04:22.304355Z","created":"2024-09-17T17:03:55.840004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-17T17:04:28.801 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2024-09-17T17:04:28.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-17T17:04:28.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-17T17:04:28.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-17T17:04:28.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-17T17:04:28.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-17T17:04:28.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:51: TEST_stretched_cluster_uneven_weight: ceph mon set election_strategy connectivity 2024-09-17T17:04:35.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:52: TEST_stretched_cluster_uneven_weight: ceph mon add disallowed_leader e 2024-09-17T17:04:40.845 INFO:tasks.workunit.client.0.smithi037.stderr:mon.e is already disallowed 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:54: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight x 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/x 2024-09-17T17:04:40.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:04:41.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:41.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:41.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/x '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:44.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:55: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight y 2024-09-17T17:04:44.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:44.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:04:44.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=y 2024-09-17T17:04:44.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:04:44.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/y 2024-09-17T17:04:44.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:04:44.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:04:44.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:44.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:44.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:44.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:44.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:44.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:44.768 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:44.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/y '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:44.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:56: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight z 2024-09-17T17:04:44.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=z 2024-09-17T17:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/z 2024-09-17T17:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:04:44.999 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:04:45.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:45.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:45.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:45.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:45.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:45.001 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:45.001 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:45.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id z --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/z '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:04:48.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: expr 4 - 1 2024-09-17T17:04:48.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: seq 0 3 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 0 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:04:48.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/0 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/0' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal' 2024-09-17T17:04:48.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:04:48.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:04:48.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2024-09-17T17:04:48.037 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:04:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/0 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3c70b7fd-c969-4add-97ba-fd055bc1d4ab 2024-09-17T17:04:48.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3c70b7fd-c969-4add-97ba-fd055bc1d4ab' 2024-09-17T17:04:48.040 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 3c70b7fd-c969-4add-97ba-fd055bc1d4ab 2024-09-17T17:04:48.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:04:48.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwtulmOFYBAxAALASlVd5bOhSMuIJYWpeosQ== 2024-09-17T17:04:48.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwtulmOFYBAxAALASlVd5bOhSMuIJYWpeosQ=="}' 2024-09-17T17:04:48.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3c70b7fd-c969-4add-97ba-fd055bc1d4ab -i td/mon-stretched-cluster-uneven-weight/0/new.json 2024-09-17T17:04:48.503 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:04:48.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/0/new.json 2024-09-17T17:04:48.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/0 --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwtulmOFYBAxAALASlVd5bOhSMuIJYWpeosQ== --osd-uuid 3c70b7fd-c969-4add-97ba-fd055bc1d4ab 2024-09-17T17:04:48.544 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:48.542+0000 7f7f91011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:48.563 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:48.561+0000 7f7f91011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:48.564 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:48.562+0000 7f7f91011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:48.564 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:48.563+0000 7f7f91011640 -1 bdev(0x55a75168fc00 td/mon-stretched-cluster-uneven-weight/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:04:48.564 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:48.563+0000 7f7f91011640 -1 bluestore(td/mon-stretched-cluster-uneven-weight/0) _read_fsid unparsable uuid 2024-09-17T17:04:57.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/0/keyring 2024-09-17T17:04:57.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:04:57.139 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-17T17:04:57.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:04:57.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:04:57.829 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-17T17:04:57.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:04:57.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/0 --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:04:57.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:04:57.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:04:57.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:04:57.856 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:57.854+0000 7f42aae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:57.858 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:57.856+0000 7f42aae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:57.859 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:57.857+0000 7f42aae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:04:58.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:04:58.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:04:58.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:04:58.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:04:58.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:04:58.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:04:58.467 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:04:58.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:04:58.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:04:58.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:04:59.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:04:59.206 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:04:59.205+0000 7f42aae0d640 -1 Falling back to public interface 2024-09-17T17:05:00.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:00.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:00.139 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:05:00.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:05:00.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:00.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:00.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:01.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:01.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:01.557 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:05:01.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:05:01.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:01.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:02.103 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:02.101+0000 7f42aae0d640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:05:04.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:05.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:05.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:05.982 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:05:05.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:05:05.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:05.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:06.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:07.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:07.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:07.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:05:07.581 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:05:07.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:07.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:08.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:09.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:09.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:09.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:05:09.084 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:05:09.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:09.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:09.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:10.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:10.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:10.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:05:10.783 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:05:10.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:10.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:11.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:12.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:12.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:12.302 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:05:12.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:05:12.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:12.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:13.155 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:14.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:14.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:14.157 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:05:14.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:05:14.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:14.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:15.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:16.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:16.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:16.121 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:05:16.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:05:16.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:16.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:16.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:17.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:17.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:17.911 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:05:17.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:05:17.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:17.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:18.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:19.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:19.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:19.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:05:19.882 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T17:05:19.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:19.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:21.255 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:22.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:22.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:22.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:05:22.257 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T17:05:22.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:22.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:24.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:25.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:25.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:25.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:05:25.111 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T17:05:25.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:25.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:26.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:27.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:27.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:27.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:05:27.961 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T17:05:27.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:27.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:29.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:30.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:30.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:30.807 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T17:05:30.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:05:30.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:30.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:05:31.408 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/139616044,v1:127.0.0.1:6803/139616044] [v2:127.0.0.1:6804/139616044,v1:127.0.0.1:6805/139616044] exists,up 3c70b7fd-c969-4add-97ba-fd055bc1d4ab 2024-09-17T17:05:31.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:05:31.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:05:31.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:05:31.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T17:05:31.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 1 2024-09-17T17:05:31.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:05:31.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/1 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:05:31.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:05:31.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:05:31.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/1' 2024-09-17T17:05:31.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal' 2024-09-17T17:05:31.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:05:31.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:05:31.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2024-09-17T17:05:31.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:05:31.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:05:31.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/1 2024-09-17T17:05:31.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:05:31.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=21e07662-7532-4712-a971-5bcbd40edb8b 2024-09-17T17:05:31.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 21e07662-7532-4712-a971-5bcbd40edb8b' 2024-09-17T17:05:31.420 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 21e07662-7532-4712-a971-5bcbd40edb8b 2024-09-17T17:05:31.420 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:05:31.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbtulm5dN0GRAAI9/mR4i9af2kVcDANA7JbQ== 2024-09-17T17:05:31.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbtulm5dN0GRAAI9/mR4i9af2kVcDANA7JbQ=="}' 2024-09-17T17:05:31.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 21e07662-7532-4712-a971-5bcbd40edb8b -i td/mon-stretched-cluster-uneven-weight/1/new.json 2024-09-17T17:05:32.783 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:05:32.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/1/new.json 2024-09-17T17:05:32.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/1 --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbtulm5dN0GRAAI9/mR4i9af2kVcDANA7JbQ== --osd-uuid 21e07662-7532-4712-a971-5bcbd40edb8b 2024-09-17T17:05:32.825 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:32.822+0000 7f65b002f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:32.827 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:32.824+0000 7f65b002f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:32.828 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:32.825+0000 7f65b002f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:32.828 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:32.826+0000 7f65b002f640 -1 bdev(0x5572d698dc00 td/mon-stretched-cluster-uneven-weight/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:05:32.828 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:32.826+0000 7f65b002f640 -1 bluestore(td/mon-stretched-cluster-uneven-weight/1) _read_fsid unparsable uuid 2024-09-17T17:05:39.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/1/keyring 2024-09-17T17:05:39.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:05:39.609 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-17T17:05:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:05:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:05:40.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:05:40.299 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-17T17:05:40.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/1 --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:05:40.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:05:40.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:05:40.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:05:40.325 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:40.323+0000 7fe09db02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:40.327 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:40.325+0000 7fe09db02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:40.329 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:40.326+0000 7fe09db02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:05:40.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:05:40.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:05:40.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:05:40.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:05:40.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:05:40.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:40.921 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:05:40.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:05:40.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:40.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:41.429 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:41.426+0000 7fe09db02640 -1 Falling back to public interface 2024-09-17T17:05:41.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:42.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:42.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:42.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:05:42.551 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:05:42.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:42.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:43.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:44.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:44.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:44.140 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:05:44.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:05:44.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:44.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:44.282 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:05:44.279+0000 7fe09db02640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:05:44.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:45.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:45.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:45.852 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:05:45.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:05:45.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:45.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:46.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:47.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:47.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:47.925 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:05:47.926 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:05:47.926 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:47.926 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:48.660 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:49.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:49.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:49.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:05:49.662 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:05:49.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:49.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:50.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:51.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:51.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:51.722 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:05:51.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:05:51.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:51.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:52.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:53.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:53.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:53.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:05:53.667 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:05:53.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:53.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:54.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:55.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:55.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:55.871 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:05:55.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:05:55.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:55.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:57.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:05:58.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:05:58.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:05:58.203 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:05:58.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:05:58.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:05:58.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:05:59.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:00.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:00.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:00.499 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:06:00.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:06:00.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:00.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:01.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:02.716 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:02.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:02.717 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T17:06:02.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:06:02.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:02.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:03.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:04.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:04.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:04.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:06:04.426 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T17:06:04.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:04.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:06.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:07.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:07.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:07.205 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T17:06:07.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:06:07.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:07.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:08.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:09.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:09.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:09.813 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T17:06:09.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:06:09.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:09.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:11.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:12.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:12.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:12.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:06:12.778 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T17:06:12.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:12.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:06:13.781 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1904640038,v1:127.0.0.1:6811/1904640038] [v2:127.0.0.1:6812/1904640038,v1:127.0.0.1:6813/1904640038] exists,up 21e07662-7532-4712-a971-5bcbd40edb8b 2024-09-17T17:06:13.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:06:13.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:06:13.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:06:13.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 2 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/2 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/2' 2024-09-17T17:06:13.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/2 2024-09-17T17:06:13.787 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:06:13.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea507bfe-0399-4151-958c-17d85b98cce4 2024-09-17T17:06:13.788 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 ea507bfe-0399-4151-958c-17d85b98cce4 2024-09-17T17:06:13.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ea507bfe-0399-4151-958c-17d85b98cce4' 2024-09-17T17:06:13.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:06:13.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFt+lml0vDLxAAeiT5hoB32u+AeyMZJK8AXQ== 2024-09-17T17:06:13.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFt+lml0vDLxAAeiT5hoB32u+AeyMZJK8AXQ=="}' 2024-09-17T17:06:13.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea507bfe-0399-4151-958c-17d85b98cce4 -i td/mon-stretched-cluster-uneven-weight/2/new.json 2024-09-17T17:06:14.769 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:06:14.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/2/new.json 2024-09-17T17:06:14.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/2 --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFt+lml0vDLxAAeiT5hoB32u+AeyMZJK8AXQ== --osd-uuid ea507bfe-0399-4151-958c-17d85b98cce4 2024-09-17T17:06:14.811 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:14.809+0000 7fdc4311b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:14.850 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:14.848+0000 7fdc4311b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:14.851 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:14.849+0000 7fdc4311b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:14.851 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:14.850+0000 7fdc4311b640 -1 bdev(0x55fe67611c00 td/mon-stretched-cluster-uneven-weight/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:06:14.851 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:14.850+0000 7fdc4311b640 -1 bluestore(td/mon-stretched-cluster-uneven-weight/2) _read_fsid unparsable uuid 2024-09-17T17:06:21.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/2/keyring 2024-09-17T17:06:21.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:06:21.827 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-17T17:06:21.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:06:21.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:06:22.493 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-17T17:06:22.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:06:22.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/2 --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:06:22.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:06:22.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:06:22.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:06:22.520 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:22.519+0000 7f9c7373e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:22.522 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:22.521+0000 7f9c7373e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:22.523 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:22.522+0000 7f9c7373e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:23.409 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:06:23.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:06:23.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:23.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:23.625 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:23.624+0000 7f9c7373e640 -1 Falling back to public interface 2024-09-17T17:06:24.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:25.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:25.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:25.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:06:25.181 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:06:25.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:25.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:25.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:26.602 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:26.600+0000 7f9c7373e640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:06:26.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:26.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:26.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:06:26.913 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:06:26.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:26.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:27.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:28.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:28.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:28.515 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:06:28.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:06:28.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:28.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:29.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:30.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:30.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:30.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:06:30.498 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:06:30.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:30.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:31.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:32.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:32.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:32.773 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:06:32.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:06:32.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:32.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:33.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:34.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:34.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:34.746 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:06:34.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:06:34.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:34.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:35.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:36.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:36.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:36.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:06:36.422 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:06:36.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:36.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:37.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:38.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:38.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:38.172 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:06:38.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:06:38.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:38.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:39.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:40.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:40.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:40.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:06:40.236 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:06:40.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:40.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:41.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:42.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:42.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:42.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:06:42.474 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:06:42.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:42.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:43.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:44.343 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T17:06:44.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:06:44.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:44.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:46.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:47.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:47.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:47.102 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T17:06:47.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:06:47.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:47.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:48.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:49.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:49.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:49.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:06:49.048 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T17:06:49.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:49.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:50.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:51.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:51.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:51.545 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T17:06:51.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:06:51.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:51.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:54.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:06:55.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:06:55.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:06:55.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:06:55.121 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T17:06:55.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:06:55.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:06:58.896 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/505506850,v1:127.0.0.1:6819/505506850] [v2:127.0.0.1:6820/505506850,v1:127.0.0.1:6821/505506850] exists,up ea507bfe-0399-4151-958c-17d85b98cce4 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 3 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T17:06:58.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/3 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/3' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:06:58.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:06:58.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/3 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d1201ec3-0544-44f9-8c0b-6955129e94db 2024-09-17T17:06:58.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 d1201ec3-0544-44f9-8c0b-6955129e94db' 2024-09-17T17:06:58.904 INFO:tasks.workunit.client.0.smithi037.stdout:add osd3 d1201ec3-0544-44f9-8c0b-6955129e94db 2024-09-17T17:06:58.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:06:58.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyt+lmyHWoNhAAfjfr7txwaVdcbiPJASZ1UQ== 2024-09-17T17:06:58.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyt+lmyHWoNhAAfjfr7txwaVdcbiPJASZ1UQ=="}' 2024-09-17T17:06:58.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d1201ec3-0544-44f9-8c0b-6955129e94db -i td/mon-stretched-cluster-uneven-weight/3/new.json 2024-09-17T17:06:59.758 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:06:59.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/3/new.json 2024-09-17T17:06:59.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/3 --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyt+lmyHWoNhAAfjfr7txwaVdcbiPJASZ1UQ== --osd-uuid d1201ec3-0544-44f9-8c0b-6955129e94db 2024-09-17T17:06:59.800 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:59.796+0000 7f8da5811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:59.813 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:59.810+0000 7f8da5811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:59.814 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:59.811+0000 7f8da5811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:06:59.814 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:59.812+0000 7f8da5811640 -1 bdev(0x55e8a4d2fc00 td/mon-stretched-cluster-uneven-weight/3/block) open stat got: (1) Operation not permitted 2024-09-17T17:06:59.814 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:06:59.812+0000 7f8da5811640 -1 bluestore(td/mon-stretched-cluster-uneven-weight/3) _read_fsid unparsable uuid 2024-09-17T17:07:06.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/3/keyring 2024-09-17T17:07:06.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:07:06.804 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd3 key to auth repository 2024-09-17T17:07:06.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T17:07:06.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:07:07.472 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.3 2024-09-17T17:07:07.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T17:07:07.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=bbedb450-ae41-45e6-a933-4e20c134f4c7 --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/3 --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.47588/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:07:07.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:07:07.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:07:07.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:07:07.499 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:07:07.497+0000 7f9a0a0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:07.502 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:07:07.499+0000 7f9a0a0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:07.503 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:07:07.500+0000 7f9a0a0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:07:08.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T17:07:08.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:07:08.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T17:07:08.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:07:08.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:07:08.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:08.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:07:08.239 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-17T17:07:08.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:08.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:08.613 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:07:08.612+0000 7f9a0a0b6640 -1 Falling back to public interface 2024-09-17T17:07:08.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:09.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:09.893 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-17T17:07:09.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:09.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:07:09.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:09.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:10.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:11.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:11.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:11.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:07:11.501 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-17T17:07:11.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:11.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:11.628 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-17T17:07:11.626+0000 7f9a0a0b6640 -1 osd.3 0 log_to_monitors true 2024-09-17T17:07:12.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:13.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:13.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:13.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:07:13.270 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-17T17:07:13.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:13.271 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:14.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:15.054 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:15.054 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:15.055 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-17T17:07:15.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:07:15.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:15.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:15.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:16.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:16.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:16.801 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-17T17:07:16.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:07:16.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:16.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:17.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:18.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:18.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:18.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:07:18.770 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-17T17:07:18.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:18.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:19.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:20.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:20.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:20.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:07:20.656 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-17T17:07:20.657 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:20.657 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:21.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:22.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:22.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:22.579 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-17T17:07:22.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:07:22.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:22.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:23.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:24.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:24.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:24.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:07:24.283 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-17T17:07:24.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:24.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:25.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:26.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:26.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:26.276 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-17T17:07:26.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:07:26.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:26.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:26.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:27.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:27.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:27.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:07:27.769 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-17T17:07:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:30.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:31.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:31.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:31.300 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-17T17:07:31.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:07:31.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:31.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:33.454 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:34.455 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:34.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:34.456 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-17T17:07:34.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:07:34.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:34.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:35.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:36.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:36.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:36.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:07:36.494 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-17T17:07:36.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:36.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:38.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:07:39.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:07:39.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:07:39.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:07:39.959 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-17T17:07:39.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:07:39.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T17:07:41.015 INFO:tasks.workunit.client.0.smithi037.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2590366913,v1:127.0.0.1:6827/2590366913] [v2:127.0.0.1:6828/2590366913,v1:127.0.0.1:6829/2590366913] exists,up d1201ec3-0544-44f9-8c0b-6955129e94db 2024-09-17T17:07:41.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:07:41.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:07:41.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:07:41.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:63: TEST_stretched_cluster_uneven_weight: for zone in iris pze 2024-09-17T17:07:41.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:65: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket iris zone 2024-09-17T17:07:44.911 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'iris' already exists 2024-09-17T17:07:44.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:66: TEST_stretched_cluster_uneven_weight: ceph osd crush move iris root=default 2024-09-17T17:07:45.482 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -5 name 'iris' to location {root=default} in crush map 2024-09-17T17:07:45.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:63: TEST_stretched_cluster_uneven_weight: for zone in iris pze 2024-09-17T17:07:45.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:65: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket pze zone 2024-09-17T17:07:46.598 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'pze' already exists 2024-09-17T17:07:46.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:66: TEST_stretched_cluster_uneven_weight: ceph osd crush move pze root=default 2024-09-17T17:07:47.563 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -7 name 'pze' to location {root=default} in crush map 2024-09-17T17:07:47.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:69: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-2 host 2024-09-17T17:07:48.774 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-2' already exists 2024-09-17T17:07:48.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:70: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-3 host 2024-09-17T17:07:49.689 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-3' already exists 2024-09-17T17:07:49.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:71: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-4 host 2024-09-17T17:07:53.683 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-4' already exists 2024-09-17T17:07:53.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:72: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-5 host 2024-09-17T17:07:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'node-5' already exists 2024-09-17T17:07:54.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:74: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-2 zone=iris 2024-09-17T17:07:56.021 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -9 name 'node-2' to location {zone=iris} in crush map 2024-09-17T17:07:56.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:75: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-3 zone=iris 2024-09-17T17:07:57.140 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -10 name 'node-3' to location {zone=iris} in crush map 2024-09-17T17:07:57.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:76: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-4 zone=pze 2024-09-17T17:07:58.555 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -11 name 'node-4' to location {zone=pze} in crush map 2024-09-17T17:07:58.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:77: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-5 zone=pze 2024-09-17T17:07:59.462 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -12 name 'node-5' to location {zone=pze} in crush map 2024-09-17T17:07:59.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:79: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.0 host=node-2 2024-09-17T17:08:00.739 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 0 name 'osd.0' to location {host=node-2} in crush map 2024-09-17T17:08:00.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:80: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.1 host=node-3 2024-09-17T17:08:01.619 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 1 name 'osd.1' to location {host=node-3} in crush map 2024-09-17T17:08:01.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:81: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.2 host=node-4 2024-09-17T17:08:02.946 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 2 name 'osd.2' to location {host=node-4} in crush map 2024-09-17T17:08:02.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:82: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.3 host=node-5 2024-09-17T17:08:04.201 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id 3 name 'osd.3' to location {host=node-5} in crush map 2024-09-17T17:08:04.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:84: TEST_stretched_cluster_uneven_weight: ceph mon set_location a zone=iris host=node-2 2024-09-17T17:08:05.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:85: TEST_stretched_cluster_uneven_weight: ceph mon set_location b zone=iris host=node-3 2024-09-17T17:08:13.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:86: TEST_stretched_cluster_uneven_weight: ceph mon set_location c zone=pze host=node-4 2024-09-17T17:08:20.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:87: TEST_stretched_cluster_uneven_weight: ceph mon set_location d zone=pze host=node-5 2024-09-17T17:08:21.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:89: TEST_stretched_cluster_uneven_weight: hostname -s 2024-09-17T17:08:21.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:89: TEST_stretched_cluster_uneven_weight: hostname=smithi037 2024-09-17T17:08:21.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:90: TEST_stretched_cluster_uneven_weight: ceph osd crush remove smithi037 2024-09-17T17:08:22.775 INFO:tasks.workunit.client.0.smithi037.stderr:device 'smithi037' does not appear in the crush map 2024-09-17T17:08:22.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:91: TEST_stretched_cluster_uneven_weight: ceph osd getcrushmap 2024-09-17T17:08:23.357 INFO:tasks.workunit.client.0.smithi037.stderr:26 2024-09-17T17:08:23.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:92: TEST_stretched_cluster_uneven_weight: crushtool --decompile crushmap 2024-09-17T17:08:23.385 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:93: TEST_stretched_cluster_uneven_weight: sed 's/^# end crush map$//' crushmap.txt 2024-09-17T17:08:23.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:94: TEST_stretched_cluster_uneven_weight: cat 2024-09-17T17:08:23.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:110: TEST_stretched_cluster_uneven_weight: crushtool --compile crushmap_modified.txt -o crushmap.bin 2024-09-17T17:08:23.402 INFO:tasks.workunit.client.0.smithi037.stderr:WARNING: min_size is no longer supported, ignoring 2024-09-17T17:08:23.402 INFO:tasks.workunit.client.0.smithi037.stderr:WARNING: max_size is no longer supported, ignoring 2024-09-17T17:08:23.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:111: TEST_stretched_cluster_uneven_weight: ceph osd setcrushmap -i crushmap.bin 2024-09-17T17:08:24.606 INFO:tasks.workunit.client.0.smithi037.stderr:28 2024-09-17T17:08:24.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:112: TEST_stretched_cluster_uneven_weight: local stretched_poolname=stretched_rbdpool 2024-09-17T17:08:24.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:113: TEST_stretched_cluster_uneven_weight: ceph osd pool create stretched_rbdpool 32 32 stretch_rule 2024-09-17T17:08:26.100 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'stretched_rbdpool' already exists 2024-09-17T17:08:26.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:114: TEST_stretched_cluster_uneven_weight: ceph osd pool set stretched_rbdpool size 4 2024-09-17T17:08:28.750 INFO:tasks.workunit.client.0.smithi037.stderr:set pool 1 size to 4 2024-09-17T17:08:28.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:116: TEST_stretched_cluster_uneven_weight: ceph mon set_location e zone=arbiter host=node-1 2024-09-17T17:08:31.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:117: TEST_stretched_cluster_uneven_weight: ceph mon enable_stretch_mode e stretch_rule zone 2024-09-17T17:08:33.122 INFO:tasks.workunit.client.0.smithi037.stderr:Second attempt of previously successful command failed with EINVAL: stretch mode is already engaged 2024-09-17T17:08:33.122 INFO:tasks.workunit.client.0.smithi037.stderr:stretch mode is already engaged 2024-09-17T17:08:33.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:120: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.09000 2024-09-17T17:08:35.151 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 0 name 'osd.0' to 0.09 in crush map 2024-09-17T17:08:35.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:121: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.1 0.09000 2024-09-17T17:08:37.142 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 1 name 'osd.1' to 0.09 in crush map 2024-09-17T17:08:37.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:122: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.2 0.09000 2024-09-17T17:08:38.903 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 2 name 'osd.2' to 0.09 in crush map 2024-09-17T17:08:38.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:123: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.3 0.09000 2024-09-17T17:08:43.401 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 3 name 'osd.3' to 0.09 in crush map 2024-09-17T17:08:43.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:126: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket sham zone 2024-09-17T17:08:44.936 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'sham' already exists 2024-09-17T17:08:44.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:127: TEST_stretched_cluster_uneven_weight: ceph osd crush move sham root=default 2024-09-17T17:08:48.506 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -3 name 'sham' to location {root=default} in crush map 2024-09-17T17:08:48.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:128: TEST_stretched_cluster_uneven_weight: wait_for_health INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:48.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1788: wait_for_health: local grepstr=INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:48.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: get_timeout_delays 300 .1 2024-09-17T17:08:48.519 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:08:48.519 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:08:48.519 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:08:48.519 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:08:48.519 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:08:48.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-17T17:08:48.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: local -a delays 2024-09-17T17:08:48.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: local -i loop=0 2024-09-17T17:08:48.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: wait_for_health: ceph health detail 2024-09-17T17:08:48.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: wait_for_health: grep INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:49.560 INFO:tasks.workunit.client.0.smithi037.stdout:[WRN] INCORRECT_NUM_BUCKETS_STRETCH_MODE: Stretch mode buckets != 2 2024-09-17T17:08:49.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:130: TEST_stretched_cluster_uneven_weight: ceph osd crush rm sham 2024-09-17T17:08:50.285 INFO:tasks.workunit.client.0.smithi037.stderr:device 'sham' does not appear in the crush map 2024-09-17T17:08:50.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:131: TEST_stretched_cluster_uneven_weight: wait_for_health_gone INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:50.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1766: wait_for_health_gone: local grepstr=INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:50.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: get_timeout_delays 300 .1 2024-09-17T17:08:50.298 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:08:50.298 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:08:50.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:08:50.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:08:50.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:08:50.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-17T17:08:50.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: local -a delays 2024-09-17T17:08:50.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health_gone: local -i loop=0 2024-09-17T17:08:50.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1770: wait_for_health_gone: ceph health detail 2024-09-17T17:08:50.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1770: wait_for_health_gone: grep INCORRECT_NUM_BUCKETS_STRETCH_MODE 2024-09-17T17:08:51.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:135: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.07000 2024-09-17T17:08:53.384 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 0 name 'osd.0' to 0.07 in crush map 2024-09-17T17:08:53.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:137: TEST_stretched_cluster_uneven_weight: wait_for_health UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:53.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1788: wait_for_health: local grepstr=UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:53.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: get_timeout_delays 300 .1 2024-09-17T17:08:53.398 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:08:53.398 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:08:53.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:08:53.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:08:53.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:08:53.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-17T17:08:53.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: wait_for_health: local -a delays 2024-09-17T17:08:53.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: local -i loop=0 2024-09-17T17:08:53.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: wait_for_health: ceph health detail 2024-09-17T17:08:53.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: wait_for_health: grep UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:54.607 INFO:tasks.workunit.client.0.smithi037.stdout:[WRN] UNEVEN_WEIGHTS_STRETCH_MODE: Stretch mode buckets have different weights! 2024-09-17T17:08:54.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:139: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.09000 2024-09-17T17:08:56.848 INFO:tasks.workunit.client.0.smithi037.stderr:reweighted item id 0 name 'osd.0' to 0.09 in crush map 2024-09-17T17:08:56.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:141: TEST_stretched_cluster_uneven_weight: wait_for_health_gone UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:56.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1766: wait_for_health_gone: local grepstr=UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:56.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: get_timeout_delays 300 .1 2024-09-17T17:08:56.864 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T17:08:56.864 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T17:08:56.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T17:08:56.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T17:08:56.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T17:08:57.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-17T17:08:57.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health_gone: local -a delays 2024-09-17T17:08:57.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health_gone: local -i loop=0 2024-09-17T17:08:57.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1770: wait_for_health_gone: ceph health detail 2024-09-17T17:08:57.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1770: wait_for_health_gone: grep UNEVEN_WEIGHTS_STRETCH_MODE 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:143: TEST_stretched_cluster_uneven_weight: teardown td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:08:57.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:08:58.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:08:58.037 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:08:58.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:08:58.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:08:58.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:08:58.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:08:58.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:08:58.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:08:58.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:08:58.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:58.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:08:58.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.120 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:08:58.120 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:58.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:08:58.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47588 2024-09-17T17:08:58.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:08:58.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:08:58.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:23: run: teardown td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:08:58.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:08:58.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:08:58.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:08:58.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:08:58.125 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:08:58.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:08:58.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:08:58.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:08:58.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:08:58.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:08:58.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:58.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:08:58.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.131 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:08:58.131 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:58.131 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:08:58.131 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47588 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/mon-stretched-cluster-uneven-weight 0 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:08:58.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:08:58.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:08:58.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:08:58.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:08:58.134 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:08:58.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:08:58.135 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:08:58.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:08:58.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:08:58.137 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:08:58.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:08:58.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:08:58.138 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:08:58.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:58.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T17:08:58.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2024-09-17T17:08:58.141 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:08:58.141 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:08:58.141 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47588 2024-09-17T17:08:58.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47588 2024-09-17T17:08:58.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:08:58.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:08:58.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T17:08:58.144 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T17:08:58.144 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T17:08:58.209 INFO:tasks.workunit:Stopping ['mon-stretch'] on client.0... 2024-09-17T17:08:58.209 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T17:08:58.558 DEBUG:teuthology.parallel:result is None 2024-09-17T17:08:58.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:08:58.583 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:08:58.583 DEBUG:teuthology.orchestra.run.smithi037:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T17:08:58.646 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T17:08:58.646 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T17:08:58.656 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T17:08:58.656 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T17:08:58.743 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T17:08:58.750 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T17:08:58.750 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:08:58.767 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +328us[ +325us] +/- 45ms 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 +186us[ +183us] +/- 59ms 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -731us[ -734us] +/- 67ms 2024-09-17T17:08:58.772 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:08:58.773 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T17:08:58.782 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T17:08:58.782 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T17:08:58.831 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T17:08:58.891 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T17:08:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:08:58.919 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T17:08:58.928 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T17:08:58.955 INFO:teuthology.task.internal:Duration was 1352.032554 seconds 2024-09-17T17:08:58.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T17:08:59.006 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T17:08:59.006 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T17:08:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:08:59.194 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T17:08:59.195 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-09-17T17:08:59.195 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T17:08:59.255 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T17:08:59.255 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T17:08:59.322 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T17:08:59.322 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T17:08:59.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T17:08:59.469 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T17:08:59.469 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T17:08:59.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T17:08:59.507 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:59.560 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-09-17T17:08:59.573 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:08:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:08:59.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T17:08:59.641 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T17:08:59.643 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909106/remote/smithi037 2024-09-17T17:08:59.644 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T17:08:59.735 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T17:08:59.736 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T17:08:59.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T17:08:59.761 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T17:08:59.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T17:08:59.790 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T17:08:59.790 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T17:08:59.807 INFO:teuthology.orchestra.run.smithi037.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 17:08 /home/ubuntu/cephtest 2024-09-17T17:08:59.808 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T17:08:59.844 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T17:08:59.946 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} duration: 1352.0325543880463 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T17:08:59.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T17:09:00.027 INFO:teuthology.run:pass