2024-09-01T01:31:49.638 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-01T01:31:49.655 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-01T01:31:49.740 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884153 branch: wip-bharath2-testing-2024-08-31-2129 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: skanta@redhat.com first_in_suite: false job_id: '7884153' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath2-testing-2024-08-31-2129 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5028887063e8eb518c95117479aa913eaa12ac8e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5028887063e8eb518c95117479aa913eaa12ac8e workunit: branch: wip-bharath2-testing-2024-08-31-2129 sha1: 5028887063e8eb518c95117479aa913eaa12ac8e owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 881 sha1: 5028887063e8eb518c95117479aa913eaa12ac8e sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath2-testing-2024-08-31-2129 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5028887063e8eb518c95117479aa913eaa12ac8e targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxpbzaRlvHfioFs3CZBpTFyvl7Oty2nzXe/s9sgc3SHuoBLZdWzFWD6J9yBRVdkWLpV8lDerkj3qZ6gi3NPrlo= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-31_23:59:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-01T01:31:49.740 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa; will attempt to use it 2024-09-01T01:31:49.741 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks 2024-09-01T01:31:49.741 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-01T01:31:49.742 INFO:teuthology.task.internal:Checking packages... 2024-09-01T01:31:49.762 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5028887063e8eb518c95117479aa913eaa12ac8e' 2024-09-01T01:31:49.762 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-01T01:31:49.762 INFO:teuthology.packaging:ref: None 2024-09-01T01:31:49.762 INFO:teuthology.packaging:tag: None 2024-09-01T01:31:49.762 INFO:teuthology.packaging:branch: wip-bharath2-testing-2024-08-31-2129 2024-09-01T01:31:49.762 INFO:teuthology.packaging:sha1: 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T01:31:49.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath2-testing-2024-08-31-2129 2024-09-01T01:31:49.900 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4640.g50288870 2024-09-01T01:31:49.902 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-01T01:31:49.908 INFO:teuthology.task.internal:no buildpackages task found 2024-09-01T01:31:49.908 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-01T01:31:49.915 INFO:teuthology.task.internal:Saving configuration 2024-09-01T01:31:49.924 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-01T01:31:49.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-01T01:31:49.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884153', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-01 01:27:36.153370', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxpbzaRlvHfioFs3CZBpTFyvl7Oty2nzXe/s9sgc3SHuoBLZdWzFWD6J9yBRVdkWLpV8lDerkj3qZ6gi3NPrlo='} 2024-09-01T01:31:49.955 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-01T01:31:49.995 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-01T01:31:49.995 INFO:teuthology.run_tasks:Running task console_log... 2024-09-01T01:31:50.030 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7d1d54bbe0>, signals=[15]) 2024-09-01T01:31:50.030 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-01T01:31:50.036 INFO:teuthology.task.internal:Opening connections... 2024-09-01T01:31:50.036 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-09-01T01:31:50.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-01T01:31:50.114 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-01T01:31:50.120 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-09-01T01:31:50.207 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-09-01T01:31:50.207 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-09-01T01:31:50.264 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-09-01T01:31:50.265 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-01T01:31:50.266 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-01T01:31:50.266 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-01T01:31:50.266 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-09-01T01:31:50.286 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-01T01:31:50.294 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-01T01:31:50.302 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-01T01:31:50.302 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-09-01T01:31:50.321 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-01T01:31:50.327 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-01T01:31:50.327 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-09-01T01:31:50.391 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-01T01:31:50.424 INFO:teuthology.run_tasks:Running task kernel... 2024-09-01T01:31:50.438 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-01T01:31:50.438 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-01T01:31:50.438 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-01T01:31:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T01:31:50.455 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-01T01:31:50.511 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-503.el9.x86_64 2024-09-01T01:31:50.511 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-503.el9.x86_64 2024-09-01T01:31:50.511 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-09-01T01:31:53.994 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-01T01:31:56.237 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-01T01:32:01.788 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-09-01T01:32:03.463 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 18 kB 00:00 2024-09-01T01:32:04.526 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 24 MB/s | 22 MB 00:00 2024-09-01T01:32:10.215 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-01T01:32:12.263 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-01T01:32:12.263 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-01T01:32:12.351 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:32:12.352 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:32:12.352 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:32:12.450 DEBUG:teuthology.orchestra.run.smithi037:> echo no | sudo yum reinstall kernel || true 2024-09-01T01:32:13.001 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:03 ago on Sun 01 Sep 2024 01:32:10 AM UTC. 2024-09-01T01:32:13.142 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:32:13.146 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-01T01:32:13.146 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.146 INFO:teuthology.orchestra.run.smithi037.stdout:Reinstalling: 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.9 M 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 0 2024-09-01T01:32:13.147 INFO:teuthology.orchestra.run.smithi037.stderr:Operation aborted. 2024-09-01T01:32:13.212 INFO:teuthology.orchestra.run.smithi037.stdout:Is this ok [y/N]: 2024-09-01T01:32:13.213 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum reinstall -y kernel || true 2024-09-01T01:32:13.759 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:03 ago on Sun 01 Sep 2024 01:32:10 AM UTC. 2024-09-01T01:32:13.892 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:32:13.892 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout:Reinstalling: 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:32:13.893 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:13.894 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.9 M 2024-09-01T01:32:13.894 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 0 2024-09-01T01:32:13.894 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:32:14.184 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-503.el9.x86_64.rpm 7.1 MB/s | 1.9 MB 00:00 2024-09-01T01:32:14.184 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:32:14.184 INFO:teuthology.orchestra.run.smithi037.stdout:Total 6.4 MB/s | 1.9 MB 00:00 2024-09-01T01:32:14.213 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:32:14.442 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:32:14.442 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:32:14.473 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:32:14.474 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:32:14.661 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:32:14.872 INFO:teuthology.orchestra.run.smithi037.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-01T01:32:14.872 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T01:32:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T01:32:15.616 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout:Reinstalled: 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:32:16.362 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:32:16.573 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-09-01T01:32:16.631 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-01T01:32:16.631 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-01T01:32:16.632 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-01T01:32:16.632 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-01T01:32:16.632 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-01T01:32:16.670 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-09-01T01:32:16.690 DEBUG:teuthology.parallel:result is None 2024-09-01T01:32:16.691 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-01T01:32:16.698 INFO:teuthology.task.internal:Creating test directory... 2024-09-01T01:32:16.698 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-01T01:32:16.746 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-01T01:32:16.753 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-01T01:32:16.762 INFO:teuthology.task.internal:Creating archive directory... 2024-09-01T01:32:16.763 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-01T01:32:16.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-01T01:32:16.816 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-01T01:32:16.816 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-01T01:32:16.897 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:32:16.907 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:32:16.909 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-01T01:32:16.916 INFO:teuthology.task.internal:Configuring sudo... 2024-09-01T01:32:16.916 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-01T01:32:16.940 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-01T01:32:16.948 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-01T01:32:16.949 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-01T01:32:16.996 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-01T01:32:17.084 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-01T01:32:17.130 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-01T01:32:17.214 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-01T01:32:17.253 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:32:17.253 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-01T01:32:17.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-09-01T01:32:17.395 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-01T01:32:17.445 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-01T01:32:17.453 INFO:teuthology.task.internal:Starting timer... 2024-09-01T01:32:17.453 INFO:teuthology.run_tasks:Running task pcp... 2024-09-01T01:32:17.510 INFO:teuthology.run_tasks:Running task selinux... 2024-09-01T01:32:17.518 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-09-01T01:32:17.571 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-09-01T01:32:17.573 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-01T01:32:17.573 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-09-01T01:32:17.594 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-09-01T01:32:17.594 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive'} 2024-09-01T01:32:17.594 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-01T01:32:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T01:32:17.660 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-01T01:32:17.660 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-09-01T01:32:17.730 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-01T01:32:17.739 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-01T01:32:17.754 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-01T01:32:17.755 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-01T01:34:01.144 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com')] 2024-09-01T01:34:01.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-01T01:34:01.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-01T01:34:01.251 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-01T01:34:01.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-01T01:34:01.341 INFO:teuthology.run_tasks:Running task clock... 2024-09-01T01:34:01.420 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-01T01:34:01.421 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-01T01:34:01.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-01T01:34:01.456 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-01T01:34:01.471 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-01T01:34:01.496 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-09-01T01:34:01.507 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-09-01T01:34:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-01T01:34:01.535 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-01T01:34:01.592 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T01:34:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T01:34:01.596 INFO:teuthology.run_tasks:Running task install... 2024-09-01T01:34:01.673 DEBUG:teuthology.task.install:project ceph 2024-09-01T01:34:01.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e'}} 2024-09-01T01:34:01.673 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e'} 2024-09-01T01:34:01.674 INFO:teuthology.task.install:Using flavor: default 2024-09-01T01:34:01.680 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-01T01:34:01.680 INFO:teuthology.task.install:extra packages: [] 2024-09-01T01:34:01.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e', 'tag': None, 'wait_for_package': False} 2024-09-01T01:34:01.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T01:34:01.828 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/ 2024-09-01T01:34:01.828 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4640.g50288870 2024-09-01T01:34:01.939 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-01T01:34:01.940 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:34:01.940 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-01T01:34:01.966 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-01T01:34:01.966 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5028887063e8eb518c95117479aa913eaa12ac8e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-01T01:34:02.033 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-01T01:34:02.108 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-01T01:34:02.172 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-09-01T01:34:02.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-09-01T01:34:02.408 INFO:teuthology.orchestra.run.smithi037.stdout:45 files removed 2024-09-01T01:34:02.435 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-09-01T01:34:03.000 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 296 kB/s | 84 kB 00:00 2024-09-01T01:34:03.293 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-01T01:34:03.568 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-01T01:34:06.301 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-01T01:34:10.311 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 20 MB 00:02 2024-09-01T01:34:15.203 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.6 MB 00:00 2024-09-01T01:34:16.904 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 18 kB 00:00 2024-09-01T01:34:17.749 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-09-01T01:34:23.397 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-01T01:34:25.792 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:34:25.794 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:25.794 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:34:25.794 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:25.794 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.3.0-4640.g50288870.el9 ceph 11 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.5 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.2 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.3.0-4640.g50288870.el9 ceph 5.5 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.3.0-4640.g50288870.el9 ceph 22 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.3.0-4640.g50288870.el9 ceph 25 k 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-01T01:34:25.795 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 755 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.3.0-4640.g50288870.el9 ceph 508 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 5.5 M 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:19.3.0-4640.g50288870.el9 ceph 45 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.3.0-4640.g50288870.el9 ceph 143 k 2024-09-01T01:34:25.796 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.3.0-4640.g50288870.el9 ceph 162 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.3.0-4640.g50288870.el9 ceph 321 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.3.0-4640.g50288870.el9 ceph 306 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.3.0-4640.g50288870.el9 ceph 99 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:34:25.797 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:25.798 INFO:teuthology.orchestra.run.smithi037.stdout:Install 30 Packages 2024-09-01T01:34:25.798 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-09-01T01:34:25.798 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:25.798 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 62 M 2024-09-01T01:34:25.798 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:34:26.949 INFO:teuthology.orchestra.run.smithi037.stdout:(1/32): ceph-base-19.3.0-4640.g50288870.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-01T01:34:27.015 INFO:teuthology.orchestra.run.smithi037.stdout:(2/32): ceph-selinux-19.3.0-4640.g50288870.el9. 380 kB/s | 25 kB 00:00 2024-09-01T01:34:27.107 INFO:teuthology.orchestra.run.smithi037.stdout:(3/32): libcephfs2-19.3.0-4640.g50288870.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-01T01:34:27.241 INFO:teuthology.orchestra.run.smithi037.stdout:(4/32): ceph-radosgw-19.3.0-4640.g50288870.el9. 14 MB/s | 11 MB 00:00 2024-09-01T01:34:27.274 INFO:teuthology.orchestra.run.smithi037.stdout:(5/32): libradosstriper1-19.3.0-4640.g50288870. 3.0 MB/s | 508 kB 00:00 2024-09-01T01:34:27.341 INFO:teuthology.orchestra.run.smithi037.stdout:(6/32): python3-ceph-argparse-19.3.0-4640.g5028 678 kB/s | 45 kB 00:00 2024-09-01T01:34:27.433 INFO:teuthology.orchestra.run.smithi037.stdout:(7/32): python3-ceph-common-19.3.0-4640.g502888 1.5 MB/s | 143 kB 00:00 2024-09-01T01:34:27.509 INFO:teuthology.orchestra.run.smithi037.stdout:(8/32): librgw2-19.3.0-4640.g50288870.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-01T01:34:27.559 INFO:teuthology.orchestra.run.smithi037.stdout:(9/32): python3-cephfs-19.3.0-4640.g50288870.el 1.3 MB/s | 162 kB 00:00 2024-09-01T01:34:27.726 INFO:teuthology.orchestra.run.smithi037.stdout:(10/32): python3-rados-19.3.0-4640.g50288870.el 1.4 MB/s | 321 kB 00:00 2024-09-01T01:34:27.751 INFO:teuthology.orchestra.run.smithi037.stdout:(11/32): python3-rbd-19.3.0-4640.g50288870.el9. 1.6 MB/s | 306 kB 00:00 2024-09-01T01:34:27.777 INFO:teuthology.orchestra.run.smithi037.stdout:(12/32): python3-rgw-19.3.0-4640.g50288870.el9. 1.9 MB/s | 99 kB 00:00 2024-09-01T01:34:28.002 INFO:teuthology.orchestra.run.smithi037.stdout:(13/32): ceph-common-19.3.0-4640.g50288870.el9. 14 MB/s | 22 MB 00:01 2024-09-01T01:34:28.144 INFO:teuthology.orchestra.run.smithi037.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 273 kB/s | 107 kB 00:00 2024-09-01T01:34:28.169 INFO:teuthology.orchestra.run.smithi037.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-09-01T01:34:28.194 INFO:teuthology.orchestra.run.smithi037.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-09-01T01:34:28.244 INFO:teuthology.orchestra.run.smithi037.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-09-01T01:34:28.320 INFO:teuthology.orchestra.run.smithi037.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-01T01:34:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-09-01T01:34:28.370 INFO:teuthology.orchestra.run.smithi037.stdout:(20/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-01T01:34:28.396 INFO:teuthology.orchestra.run.smithi037.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 475 kB/s | 36 kB 00:00 2024-09-01T01:34:28.421 INFO:teuthology.orchestra.run.smithi037.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-01T01:34:28.488 INFO:teuthology.orchestra.run.smithi037.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-01T01:34:28.513 INFO:teuthology.orchestra.run.smithi037.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 342 kB/s | 26 kB 00:00 2024-09-01T01:34:28.538 INFO:teuthology.orchestra.run.smithi037.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-01T01:34:28.564 INFO:teuthology.orchestra.run.smithi037.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-01T01:34:28.606 INFO:teuthology.orchestra.run.smithi037.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-01T01:34:28.639 INFO:teuthology.orchestra.run.smithi037.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-01T01:34:28.731 INFO:teuthology.orchestra.run.smithi037.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-01T01:34:28.831 INFO:teuthology.orchestra.run.smithi037.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-01T01:34:28.915 INFO:teuthology.orchestra.run.smithi037.stdout:(31/32): librados2-19.3.0-4640.g50288870.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-01T01:34:28.990 INFO:teuthology.orchestra.run.smithi037.stdout:(32/32): librbd1-19.3.0-4640.g50288870.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-01T01:34:28.996 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:34:28.996 INFO:teuthology.orchestra.run.smithi037.stdout:Total 19 MB/s | 62 MB 00:03 2024-09-01T01:34:29.097 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:34:29.137 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:34:29.138 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:34:29.637 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:34:29.638 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:34:31.137 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:34:31.233 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-01T01:34:31.262 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-01T01:34:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-01T01:34:31.310 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-01T01:34:31.334 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-01T01:34:31.660 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-01T01:34:31.716 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-01T01:34:32.108 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-01T01:34:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T01:34:32.257 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T01:34:32.347 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 8/34 2024-09-01T01:34:32.374 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 9/34 2024-09-01T01:34:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 9/34 2024-09-01T01:34:32.466 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-01T01:34:32.510 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.3.0-4640.g50288870.el9. 11/34 2024-09-01T01:34:32.589 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 12/34 2024-09-01T01:34:32.618 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 13/34 2024-09-01T01:34:32.682 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 13/34 2024-09-01T01:34:32.758 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-01T01:34:32.868 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-01T01:34:32.900 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-01T01:34:32.941 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-01T01:34:32.966 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-01T01:34:32.995 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-01T01:34:33.028 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-01T01:34:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-01T01:34:33.407 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.3.0-4640.g50288870.el9.x86_64 22/34 2024-09-01T01:34:33.490 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.3.0-4640.g50288870.el9.x86_64 22/34 2024-09-01T01:34:33.520 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 23/34 2024-09-01T01:34:33.564 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-01T01:34:34.100 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-01T01:34:34.126 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.3.0-4640.g50288870.el9.x86_64 26/34 2024-09-01T01:34:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.3.0-4640.g50288870.el9.x86_64 26/34 2024-09-01T01:34:34.269 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 27/34 2024-09-01T01:34:34.303 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.3.0-4640.g50288870.el9.x8 28/34 2024-09-01T01:34:36.758 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T01:34:36.809 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T01:34:37.391 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T01:34:37.428 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 30/34 2024-09-01T01:34:37.519 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 30/34 2024-09-01T01:34:37.519 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-01T01:34:37.519 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-01T01:34:37.519 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:37.548 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T01:34:48.085 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-09-01T01:34:48.086 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:49.114 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 32/34 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 32/34 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-01T01:34:49.140 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:49.142 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-01T01:34:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-01T01:34:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T01:34:50.497 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 1/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 3/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 4/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 5/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 6/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4640.g50288870.el9. 8/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.3.0-4640.g50288870.el9.x8 9/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 10/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 11/34 2024-09-01T01:34:50.498 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 12/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 13/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-01T01:34:50.499 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-01T01:34:50.500 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.3.0-4640.g50288870.el9.x86_64 33/34 2024-09-01T01:34:51.308 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-01T01:34:51.309 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.310 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-20.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:51.311 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:34:51.560 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-09-01T01:34:52.135 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:29 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:34:52.272 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.3.0-4640.g50288870.el9 ceph 50 M 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.3.0-4640.g50288870.el9 ceph 161 k 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:52.273 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 51 M 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 216 M 2024-09-01T01:34:52.274 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:34:52.675 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-01T01:34:52.725 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-01T01:34:52.784 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): libcephsqlite-19.3.0-4640.g50288870.el9. 635 kB/s | 161 kB 00:00 2024-09-01T01:34:54.105 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): ceph-test-19.3.0-4640.g50288870.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-01T01:34:54.106 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:34:54.106 INFO:teuthology.orchestra.run.smithi037.stdout:Total 28 MB/s | 51 MB 00:01 2024-09-01T01:34:54.132 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:34:54.154 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:34:54.155 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:34:54.550 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:34:54.551 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:34:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:34:55.213 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-01T01:34:55.252 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-01T01:34:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 3/4 2024-09-01T01:35:00.232 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 3/4 2024-09-01T01:35:00.313 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 4/4 2024-09-01T01:35:02.742 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 4/4 2024-09-01T01:35:02.756 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 1/4 2024-09-01T01:35:02.756 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 2/4 2024-09-01T01:35:02.757 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:03.076 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-01T01:35:03.077 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-01T01:35:03.077 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:03.077 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:03.369 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-09-01T01:35:03.939 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:40 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:04.081 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.3.0-4640.g50288870.el9 ceph 6.6 k 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.3.0-4640.g50288870.el9 ceph 2.4 M 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.3.0-4640.g50288870.el9 ceph 1.5 M 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 290 k 2024-09-01T01:35:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.3.0-4640.g50288870.el9 ceph 4.8 M 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.3.0-4640.g50288870.el9 ceph 17 M 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections 2024-09-01T01:35:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-01T01:35:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout:Install 37 Packages 2024-09-01T01:35:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:04.088 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 M 2024-09-01T01:35:04.088 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 109 M 2024-09-01T01:35:04.088 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:04.702 INFO:teuthology.orchestra.run.smithi037.stdout:(1/37): ceph-19.3.0-4640.g50288870.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-01T01:35:04.853 INFO:teuthology.orchestra.run.smithi037.stdout:(2/37): ceph-mgr-19.3.0-4640.g50288870.el9.x86_ 5.3 MB/s | 1.5 MB 00:00 2024-09-01T01:35:04.920 INFO:teuthology.orchestra.run.smithi037.stdout:(3/37): ceph-mds-19.3.0-4640.g50288870.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-01T01:35:04.987 INFO:teuthology.orchestra.run.smithi037.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4640.g5028 4.3 MB/s | 290 kB 00:00 2024-09-01T01:35:05.154 INFO:teuthology.orchestra.run.smithi037.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-01T01:35:05.296 INFO:teuthology.orchestra.run.smithi037.stdout:(6/37): ceph-mon-19.3.0-4640.g50288870.el9.x86_ 8.1 MB/s | 4.8 MB 00:00 2024-09-01T01:35:05.329 INFO:teuthology.orchestra.run.smithi037.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.0 MB/s | 1.2 MB 00:00 2024-09-01T01:35:05.513 INFO:teuthology.orchestra.run.smithi037.stdout:(8/37): ceph-osd-19.3.0-4640.g50288870.el9.x86_ 26 MB/s | 17 MB 00:00 2024-09-01T01:35:05.547 INFO:teuthology.orchestra.run.smithi037.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 623 kB/s | 135 kB 00:00 2024-09-01T01:35:05.572 INFO:teuthology.orchestra.run.smithi037.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 386 kB/s | 106 kB 00:00 2024-09-01T01:35:05.597 INFO:teuthology.orchestra.run.smithi037.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-01T01:35:05.647 INFO:teuthology.orchestra.run.smithi037.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-01T01:35:05.673 INFO:teuthology.orchestra.run.smithi037.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 783 kB/s | 125 kB 00:00 2024-09-01T01:35:05.698 INFO:teuthology.orchestra.run.smithi037.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 687 kB/s | 35 kB 00:00 2024-09-01T01:35:05.740 INFO:teuthology.orchestra.run.smithi037.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-01T01:35:05.765 INFO:teuthology.orchestra.run.smithi037.stdout:(16/37): lua-5.4.4-4.el9.x86_64.rpm 975 kB/s | 188 kB 00:00 2024-09-01T01:35:05.790 INFO:teuthology.orchestra.run.smithi037.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-09-01T01:35:05.816 INFO:teuthology.orchestra.run.smithi037.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-01T01:35:05.866 INFO:teuthology.orchestra.run.smithi037.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-01T01:35:05.891 INFO:teuthology.orchestra.run.smithi037.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-01T01:35:05.916 INFO:teuthology.orchestra.run.smithi037.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-01T01:35:05.942 INFO:teuthology.orchestra.run.smithi037.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-01T01:35:05.967 INFO:teuthology.orchestra.run.smithi037.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-01T01:35:05.992 INFO:teuthology.orchestra.run.smithi037.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 924 kB/s | 23 kB 00:00 2024-09-01T01:35:06.017 INFO:teuthology.orchestra.run.smithi037.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-01T01:35:06.043 INFO:teuthology.orchestra.run.smithi037.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-01T01:35:06.068 INFO:teuthology.orchestra.run.smithi037.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-01T01:35:06.093 INFO:teuthology.orchestra.run.smithi037.stdout:(28/37): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-09-01T01:35:06.135 INFO:teuthology.orchestra.run.smithi037.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 503 kB/s | 46 kB 00:00 2024-09-01T01:35:06.169 INFO:teuthology.orchestra.run.smithi037.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2024-09-01T01:35:06.194 INFO:teuthology.orchestra.run.smithi037.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-09-01T01:35:06.219 INFO:teuthology.orchestra.run.smithi037.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-01T01:35:06.245 INFO:teuthology.orchestra.run.smithi037.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-01T01:35:06.270 INFO:teuthology.orchestra.run.smithi037.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-01T01:35:06.295 INFO:teuthology.orchestra.run.smithi037.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-01T01:35:06.320 INFO:teuthology.orchestra.run.smithi037.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-01T01:35:06.354 INFO:teuthology.orchestra.run.smithi037.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-01T01:35:06.359 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:06.359 INFO:teuthology.orchestra.run.smithi037.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-01T01:35:06.511 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:06.555 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:06.555 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:06.869 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:06.869 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:08.265 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:08.332 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-01T01:35:08.365 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-01T01:35:08.393 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-01T01:35:08.467 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-01T01:35:08.509 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-01T01:35:08.550 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-01T01:35:08.656 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-01T01:35:08.741 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-01T01:35:08.786 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-01T01:35:08.821 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-01T01:35:08.850 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-01T01:35:08.885 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-01T01:35:08.918 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-01T01:35:08.967 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-01T01:35:09.019 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-01T01:35:09.070 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-01T01:35:09.119 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-01T01:35:09.159 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-01T01:35:09.234 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-01T01:35:09.379 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-01T01:35:09.462 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-01T01:35:09.521 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-01T01:35:09.559 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-01T01:35:09.616 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-01T01:35:09.681 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-01T01:35:09.879 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-01T01:35:09.938 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-01T01:35:10.019 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-01T01:35:10.115 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-01T01:35:10.153 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-01T01:35:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-01T01:35:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9. 32/37 2024-09-01T01:35:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 33/37 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 33/37 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-01T01:35:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:12.063 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 34/37 2024-09-01T01:35:12.090 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 34/37 2024-09-01T01:35:12.091 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:12.091 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-01T01:35:12.091 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-01T01:35:12.091 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-01T01:35:12.091 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:12.564 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 35/37 2024-09-01T01:35:12.587 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 35/37 2024-09-01T01:35:12.587 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:12.587 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-01T01:35:12.587 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-01T01:35:12.587 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-01T01:35:12.588 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:12.823 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 36/37 2024-09-01T01:35:12.846 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 36/37 2024-09-01T01:35:12.846 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:12.846 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-01T01:35:12.846 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-01T01:35:12.846 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-01T01:35:12.847 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:12.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.3.0-4640.g50288870.el9.x86_64 37/37 2024-09-01T01:35:14.609 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:19.3.0-4640.g50288870.el9.x86_64 37/37 2024-09-01T01:35:14.609 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.3.0-4640.g50288870.el9.x86_64 1/37 2024-09-01T01:35:14.609 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 2/37 2024-09-01T01:35:14.609 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 3/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 4/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 5/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9. 6/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-01T01:35:14.610 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-01T01:35:14.611 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-01T01:35:14.612 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-01T01:35:15.546 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-01T01:35:15.546 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:15.546 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:15.546 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:15.546 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-01T01:35:15.547 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-01T01:35:15.548 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:15.549 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:15.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-09-01T01:35:16.495 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:53 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:16.587 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:35:16.631 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:16.632 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:35:16.632 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:16.705 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-09-01T01:35:17.264 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:54 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:17.401 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 783 k 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 783 k 2024-09-01T01:35:17.402 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 789 k 2024-09-01T01:35:17.403 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:18.079 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-19.3.0-4640.g50288870.el9.noarch.rpm 1.1 MB/s | 783 kB 00:00 2024-09-01T01:35:18.080 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:18.080 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.1 MB/s | 783 kB 00:00 2024-09-01T01:35:18.080 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:18.085 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:18.085 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:18.091 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:18.092 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:18.167 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:18.633 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:18.717 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:19.116 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:19.358 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:19.358 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:19.359 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:19.359 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:19.359 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:19.359 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:19.551 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-09-01T01:35:20.112 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:57 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:20.250 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4640.g50288870.el9 ceph 145 k 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 145 k 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 432 k 2024-09-01T01:35:20.251 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:20.437 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-19.3.0-4640.g502888 783 kB/s | 145 kB 00:00 2024-09-01T01:35:20.438 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:20.438 INFO:teuthology.orchestra.run.smithi037.stdout:Total 776 kB/s | 145 kB 00:00 2024-09-01T01:35:20.438 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:20.446 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:20.446 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:20.509 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:20.509 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:20.658 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:20.764 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T01:35:20.788 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T01:35:20.788 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:20.788 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-01T01:35:20.788 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:21.711 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T01:35:21.712 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:21.712 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:21.712 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:35:21.712 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:21.712 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:21.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-09-01T01:35:22.474 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:59 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:22.566 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:35:22.610 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:22.611 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:35:22.612 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:22.690 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-09-01T01:35:23.248 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:00 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:23.385 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 14 M 2024-09-01T01:35:23.386 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 34 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 16 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 19 M 2024-09-01T01:35:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 158 M 2024-09-01T01:35:23.389 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:23.773 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4640.g50 224 kB/s | 34 kB 00:00 2024-09-01T01:35:23.798 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4640.g502 92 kB/s | 16 kB 00:00 2024-09-01T01:35:24.007 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-01T01:35:24.099 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-01T01:35:24.158 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-01T01:35:24.183 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-09-01T01:35:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4640.g5028887 20 MB/s | 14 MB 00:00 2024-09-01T01:35:24.358 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-01T01:35:24.417 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 609 kB/s | 31 kB 00:00 2024-09-01T01:35:24.442 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-09-01T01:35:24.467 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-01T01:35:24.509 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.3 MB/s | 2.0 MB 00:00 2024-09-01T01:35:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:24.513 INFO:teuthology.orchestra.run.smithi037.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-01T01:35:24.562 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:24.582 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:25.432 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:25.527 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-01T01:35:25.562 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-01T01:35:25.610 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-01T01:35:25.647 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-01T01:35:25.727 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-01T01:35:25.870 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-01T01:35:25.979 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-01T01:35:26.045 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-01T01:35:26.072 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-01T01:35:26.104 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9 10/12 2024-09-01T01:35:29.475 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el 11/12 2024-09-01T01:35:29.535 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 12/12 2024-09-01T01:35:31.509 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 12/12 2024-09-01T01:35:31.509 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el 1/12 2024-09-01T01:35:31.509 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 2/12 2024-09-01T01:35:31.509 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9 3/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-01T01:35:31.510 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-01T01:35:31.840 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:31.841 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:32.105 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-01T01:35:32.670 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:09 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:32.807 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 7.4 M 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-01T01:35:32.809 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-01T01:35:32.810 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:32.811 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 38 M 2024-09-01T01:35:32.811 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 212 M 2024-09-01T01:35:32.811 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:33.175 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 789 kB/s | 191 kB 00:00 2024-09-01T01:35:33.217 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-01T01:35:33.243 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 487 kB/s | 33 kB 00:00 2024-09-01T01:35:33.293 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-01T01:35:33.351 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 661 kB/s | 38 kB 00:00 2024-09-01T01:35:33.443 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 14 MB/s | 7.4 MB 00:00 2024-09-01T01:35:33.602 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-09-01T01:35:33.660 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-01T01:35:33.869 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-09-01T01:35:33.936 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 8.5 MB/s | 4.9 MB 00:00 2024-09-01T01:35:34.103 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-09-01T01:35:34.886 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-09-01T01:35:34.887 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:34.887 INFO:teuthology.orchestra.run.smithi037.stdout:Total 18 MB/s | 38 MB 00:02 2024-09-01T01:35:35.250 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:35.276 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:35.276 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:35.710 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:35.710 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:36.487 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-01T01:35:36.593 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-01T01:35:36.696 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-01T01:35:36.736 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-01T01:35:36.926 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-01T01:35:36.951 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-01T01:35:37.029 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-01T01:35:37.709 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-01T01:35:37.844 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-01T01:35:39.137 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-01T01:35:40.214 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-01T01:35:40.241 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 12/12 2024-09-01T01:35:42.766 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 12/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 1/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-01T01:35:42.767 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-01T01:35:43.165 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-01T01:35:43.165 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:43.166 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:43.555 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-09-01T01:35:44.122 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:21 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 49 k 2024-09-01T01:35:44.261 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:44.262 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.2 M 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-09-01T01:35:44.263 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:45.108 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): ceph-mgr-rook-19.3.0-4640.g50288870.el9 281 kB/s | 49 kB 00:00 2024-09-01T01:35:45.467 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 36 kB/s | 19 kB 00:00 2024-09-01T01:35:45.492 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-01T01:35:45.876 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 290 kB/s | 222 kB 00:00 2024-09-01T01:35:46.184 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-01T01:35:46.234 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 661 kB/s | 32 kB 00:00 2024-09-01T01:35:46.260 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 568 kB/s | 14 kB 00:00 2024-09-01T01:35:46.293 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 348 kB/s | 279 kB 00:00 2024-09-01T01:35:46.318 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.7 MB/s | 220 kB 00:00 2024-09-01T01:35:46.377 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 12 MB/s | 1.0 MB 00:00 2024-09-01T01:35:46.402 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 708 kB/s | 59 kB 00:00 2024-09-01T01:35:46.428 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-01T01:35:46.494 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-01T01:35:46.498 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:46.498 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.0 MB/s | 2.2 MB 00:02 2024-09-01T01:35:46.569 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:46.586 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:46.587 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:46.680 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:46.680 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:47.186 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:47.367 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-01T01:35:47.423 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-01T01:35:47.479 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-01T01:35:47.513 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-01T01:35:47.556 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-01T01:35:47.672 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-01T01:35:47.790 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-01T01:35:47.841 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-01T01:35:48.599 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-01T01:35:48.654 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-01T01:35:48.696 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-01T01:35:48.745 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-01T01:35:48.773 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 13/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 13/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 1/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-01T01:35:49.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-01T01:35:49.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-01T01:35:49.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-01T01:35:49.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-01T01:35:49.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-01T01:35:49.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-01T01:35:49.879 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:49.880 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:50.150 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-09-01T01:35:50.716 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:27 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:50.854 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 154 k 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-01T01:35:50.855 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 7.0 M 2024-09-01T01:35:50.856 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-09-01T01:35:50.857 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:51.329 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-01T01:35:51.387 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-01T01:35:51.413 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-01T01:35:51.438 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4640.g50288870.e 476 kB/s | 154 kB 00:00 2024-09-01T01:35:51.463 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-01T01:35:51.922 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 6.0 MB 00:00 2024-09-01T01:35:51.924 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:51.924 INFO:teuthology.orchestra.run.smithi037.stdout:Total 6.6 MB/s | 7.0 MB 00:01 2024-09-01T01:35:51.996 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:52.008 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:52.008 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:52.111 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:52.111 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:52.427 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:52.620 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-01T01:35:52.675 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-01T01:35:53.224 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-01T01:35:53.313 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-01T01:35:53.389 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-01T01:35:53.419 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 6/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 6/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-01T01:35:54.161 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-01T01:35:54.471 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-01T01:35:54.471 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:54.471 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:54.471 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:35:54.471 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:54.472 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:54.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-09-01T01:35:55.309 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:32 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.3.0-4640.g50288870.el9 ceph 900 k 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 979 k 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.9 M 2024-09-01T01:35:55.449 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:55.856 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-01T01:35:55.973 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-fuse-19.3.0-4640.g50288870.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-01T01:35:55.974 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:55.974 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.8 MB/s | 979 kB 00:00 2024-09-01T01:35:55.986 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:56.010 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:56.010 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:56.095 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:56.095 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:56.295 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:56.455 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-01T01:35:56.523 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2/2 2024-09-01T01:35:57.159 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2/2 2024-09-01T01:35:57.159 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/2 2024-09-01T01:35:57.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-01T01:35:57.420 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:57.420 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:35:57.420 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-01T01:35:57.421 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:57.421 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:35:57.703 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-volume 2024-09-01T01:35:58.274 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:35 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 285 k 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 285 k 2024-09-01T01:35:58.416 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.3 M 2024-09-01T01:35:58.416 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:35:58.644 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-volume-19.3.0-4640.g50288870.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-01T01:35:58.644 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:35:58.645 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-01T01:35:58.645 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:35:58.654 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:35:58.654 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:35:58.677 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:35:58.677 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:35:58.901 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:35:58.979 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:58.995 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:35:58.995 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:35:58.995 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-01T01:35:58.995 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:00.003 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:00.300 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-09-01T01:36:00.870 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:37 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:01.008 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.3.0-4640.g50288870.el9 ceph 126 k 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:01.009 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 126 k 2024-09-01T01:36:01.010 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-09-01T01:36:01.010 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:36:01.215 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-19.3.0-4640.g50288870.el9.x86_64 617 kB/s | 126 kB 00:00 2024-09-01T01:36:01.215 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:36:01.216 INFO:teuthology.orchestra.run.smithi037.stdout:Total 613 kB/s | 126 kB 00:00 2024-09-01T01:36:01.216 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:36:01.222 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:36:01.222 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:36:01.288 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:36:01.288 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:36:01.445 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:36:01.547 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:02.244 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:02.244 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:02.244 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:02.244 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:36:02.245 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:02.245 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:02.483 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-09-01T01:36:03.055 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:03.149 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:03.195 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:03.196 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:03.196 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:03.275 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-09-01T01:36:03.838 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:03.976 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.3.0-4640.g50288870.el9 ceph 32 k 2024-09-01T01:36:03.977 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 32 k 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 142 k 2024-09-01T01:36:03.978 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:36:04.143 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-19.3.0-4640.g50288870.el9.x86_6 193 kB/s | 32 kB 00:00 2024-09-01T01:36:04.143 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:36:04.143 INFO:teuthology.orchestra.run.smithi037.stdout:Total 191 kB/s | 32 kB 00:00 2024-09-01T01:36:04.143 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:36:04.147 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:36:04.147 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:36:04.151 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:36:04.152 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:36:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:36:04.358 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:04.696 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:04.911 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:05.099 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-09-01T01:36:05.664 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:42 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:05.756 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:05.801 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:05.802 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:05.803 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:05.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-09-01T01:36:06.438 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:43 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:06.531 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:06.576 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:06.577 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:06.577 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:06.647 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-09-01T01:36:07.213 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:44 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:07.305 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:07.350 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:07.351 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:07.351 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:07.425 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-09-01T01:36:07.988 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:44 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:08.081 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:08.126 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:08.127 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:08.127 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:08.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-09-01T01:36:08.765 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:45 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:08.858 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:08.904 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:08.905 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:08.905 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:08.971 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-09-01T01:36:09.538 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:46 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:09.632 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T01:36:09.677 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:09.678 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-01T01:36:09.678 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:09.749 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-09-01T01:36:10.314 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:47 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:10.455 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:10.455 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-01T01:36:10.455 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.3.0-4640.g50288870.el9 ceph 85 k 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:10.456 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 85 k 2024-09-01T01:36:10.457 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 231 k 2024-09-01T01:36:10.457 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:36:10.646 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-19.3.0-4640.g50288870.el9.x86_64.rpm 451 kB/s | 85 kB 00:00 2024-09-01T01:36:10.647 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:36:10.647 INFO:teuthology.orchestra.run.smithi037.stdout:Total 448 kB/s | 85 kB 00:00 2024-09-01T01:36:10.647 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:36:10.654 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:36:10.655 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:36:10.721 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:36:10.721 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:36:10.872 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:36:10.992 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:11.403 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:11.660 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:36:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:11.660 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:11.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-09-01T01:36:12.526 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:49 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:12.665 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:12.665 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.2 M 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.2 M 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 14 M 2024-09-01T01:36:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:36:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-19.3.0-4640.g50288870.el9.x86_64.rpm 2.7 MB/s | 3.2 MB 00:01 2024-09-01T01:36:13.858 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:36:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.7 MB/s | 3.2 MB 00:01 2024-09-01T01:36:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:36:13.869 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:36:13.869 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:36:13.956 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:36:13.956 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:36:14.417 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:36:14.484 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:14.509 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:14.509 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T01:36:14.510 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-01T01:36:14.510 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-01T01:36:14.510 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-01T01:36:14.510 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:15.585 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:15.854 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-09-01T01:36:16.427 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:53 ago on Sun 01 Sep 2024 01:34:23 AM UTC. 2024-09-01T01:36:16.564 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.3.0-4640.g50288870.el9 ceph 171 k 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-01T01:36:16.565 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:16.566 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 171 k 2024-09-01T01:36:16.566 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 490 k 2024-09-01T01:36:16.566 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-01T01:36:16.778 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-19.3.0-4640.g50288870.el9.x86_64.rpm 809 kB/s | 171 kB 00:00 2024-09-01T01:36:16.778 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-01T01:36:16.778 INFO:teuthology.orchestra.run.smithi037.stdout:Total 804 kB/s | 171 kB 00:00 2024-09-01T01:36:16.779 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-01T01:36:16.787 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-01T01:36:16.787 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-01T01:36:16.853 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-01T01:36:16.853 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-01T01:36:17.013 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-01T01:36:17.109 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:17.531 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-01T01:36:17.775 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-01T01:36:18.047 DEBUG:teuthology.parallel:result is None 2024-09-01T01:36:18.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T01:36:18.192 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-01T01:36:18.213 INFO:teuthology.orchestra.run.smithi037.stdout:19.3.0-4640.g50288870.el9 2024-09-01T01:36:18.214 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4640.g50288870.el9 2024-09-01T01:36:18.214 INFO:teuthology.task.install:The correct ceph version 19.3.0-4640.g50288870 is installed. 2024-09-01T01:36:18.216 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-01T01:36:18.216 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:36:18.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-01T01:36:18.278 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-01T01:36:18.279 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:36:18.279 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-09-01T01:36:18.343 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-01T01:36:18.408 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-01T01:36:18.408 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:36:18.408 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-01T01:36:18.472 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-01T01:36:18.536 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-01T01:36:18.537 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:36:18.537 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-09-01T01:36:18.602 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-01T01:36:18.666 INFO:teuthology.run_tasks:Running task workunit... 2024-09-01T01:36:18.682 INFO:tasks.workunit:Pulling workunits from ref 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T01:36:18.682 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-01T01:36:18.682 INFO:tasks.workunit:timeout=3h 2024-09-01T01:36:18.682 INFO:tasks.workunit:cleanup=True 2024-09-01T01:36:18.683 DEBUG:teuthology.orchestra.run.smithi037:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-01T01:36:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T01:36:18.721 INFO:teuthology.orchestra.run.smithi037.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-01T01:36:18.721 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-01T01:36:18.778 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-01T01:36:18.778 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-01T01:36:18.834 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T01:36:18.889 INFO:tasks.workunit.client.0.smithi037.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-01T01:37:07.610 INFO:tasks.workunit.client.0.smithi037.stderr:Note: switching to '5028887063e8eb518c95117479aa913eaa12ac8e'. 2024-09-01T01:37:07.610 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.610 INFO:tasks.workunit.client.0.smithi037.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-01T01:37:07.610 INFO:tasks.workunit.client.0.smithi037.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr:state without impacting any branches by switching back to a branch. 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: git switch -c 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr:Or undo this operation with: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: git switch - 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-01T01:37:07.611 INFO:tasks.workunit.client.0.smithi037.stderr: 2024-09-01T01:37:07.612 INFO:tasks.workunit.client.0.smithi037.stderr:HEAD is now at 5028887063e Merge branch 'wip-bug-67087' of https://github.com/rzarzynski/ceph into wip-bharath2-testing-2024-08-31-2129 2024-09-01T01:37:07.614 DEBUG:teuthology.orchestra.run.smithi037:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-01T01:37:07.671 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-01T01:37:07.671 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-01T01:37:07.726 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-01T01:37:07.727 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-01T01:37:07.727 DEBUG:teuthology.orchestra.run.smithi037:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-01T01:37:07.795 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T01:37:07.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/mclock-config 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-01T01:37:07.802 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-01T01:37:07.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none ' 2024-09-01T01:37:07.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-01T01:37:07.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-01T01:37:07.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-01T01:37:07.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T01:37:07.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-01T01:37:07.813 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_profile_builtin_to_custom 2024-09-01T01:37:07.813 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_profile_custom_to_builtin 2024-09-01T01:37:07.813 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-01T01:37:07.813 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_profile_disallow_builtin_params_override 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:07.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:07.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:37:07.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:37:07.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:07.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:37:07.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:37:07.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:07.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:37:07.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:37:07.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:37:07.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:37:07.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:07.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:07.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:37:07.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:37:07.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:37:07.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:37:07.822 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:37:07.822 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:07.822 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:07.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:37:07.824 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:37:07.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T01:37:07.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:37:07.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:07.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:37:07.827 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:37:07.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:07.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:37:07.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:37:07.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:07.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:37:07.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:37:07.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:37:07.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:37:07.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:07.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:07.831 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:37:07.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:37:07.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:37:07.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:37:07.833 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:37:07.833 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:07.833 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:07.833 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:37:07.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:37:07.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:37:07.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:37:07.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:37:08.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:37:08.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:37:08.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:37:08.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:37:08.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:37:08.604 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:37:08.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:08.706 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:08.707 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:08.707 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:37:08.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:37:08.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:37:08.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:09.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:37:09.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:37:09.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:09.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:37:09.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:37:09.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:37:09.668 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:37:09.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:09.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 24053e71-2380-4e71-b9f8-cbfec990a294' 2024-09-01T01:37:09.668 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:09.669 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:37:09.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFxdNmD+y2KBAAQghNJ135EVCPrat4+ZQL7w== 2024-09-01T01:37:09.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFxdNmD+y2KBAAQghNJ135EVCPrat4+ZQL7w=="}' 2024-09-01T01:37:09.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 24053e71-2380-4e71-b9f8-cbfec990a294 -i td/mclock-config/0/new.json 2024-09-01T01:37:09.962 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:37:09.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:37:09.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBFxdNmD+y2KBAAQghNJ135EVCPrat4+ZQL7w== --osd-uuid 24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:10.016 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:10.014+0000 7fddc1218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:10.031 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:10.030+0000 7fddc1218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:10.031 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:10.031+0000 7fddc1218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:10.032 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:10.032+0000 7fddc1218640 -1 bdev(0x55982f863c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:37:10.032 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:10.032+0000 7fddc1218640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:37:16.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:37:16.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:37:16.293 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:37:16.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:37:16.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:37:16.854 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:37:16.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:37:16.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:37:16.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:37:16.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:37:16.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:37:16.880 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:16.880+0000 7ff6a4dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:16.891 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:16.891+0000 7ff6a4dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:16.892 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:16.892+0000 7ff6a4dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:17.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:37:17.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:37:17.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:37:17.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:37:17.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:37:17.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:17.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:37:17.315 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:37:17.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:17.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:17.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:18.485 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:18.485+0000 7ff6a4dcc640 -1 Falling back to public interface 2024-09-01T01:37:18.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:18.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:18.708 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:37:18.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:37:18.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:18.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:20.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:20.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:20.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:37:20.142 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:37:20.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:20.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:20.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:21.070 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:21.070+0000 7ff6a4dcc640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:37:21.552 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:21.552 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:21.552 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:37:21.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:37:21.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:21.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:22.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:23.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:23.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:23.099 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:37:23.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:37:23.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:23.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:23.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:24.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:24.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:24.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:37:24.522 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:37:24.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:24.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:24.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:25.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:25.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:25.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:37:25.936 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:37:25.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:25.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:26.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:27.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:27.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:27.350 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:37:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:37:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:27.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:28.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:28.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:28.782 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:37:28.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:37:28.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:28.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:29.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:30.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:30.211 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:37:30.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:30.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:37:30.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:30.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:30.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:31.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:31.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:31.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:37:31.621 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:37:31.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:31.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:32.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:33.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:33.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:33.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:37:33.174 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:37:33.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:33.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:33.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:34.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:34.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:34.757 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:37:34.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:37:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:35.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:36.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:36.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:36.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:37:36.545 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:37:36.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:36.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:37.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:38.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:38.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:38.506 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:37:38.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:37:38.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:38.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:39.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:40.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:40.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:40.347 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:37:40.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:37:40.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:40.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:42.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:43.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:43.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:43.120 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:37:43.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:37:43.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:43.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:44.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:45.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:45.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:45.138 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:37:45.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:37:45.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:45.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:45.701 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/510913172,v1:127.0.0.1:6803/510913172] [v2:127.0.0.1:6804/510913172,v1:127.0.0.1:6805/510913172] exists,up 24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:45.703 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:45.704 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:45.704 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:45.704 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:45.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:45.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:37:45.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-01T01:37:45.789 INFO:tasks.workunit.client.0.smithi037.stdout:osd_max_backfills: 1 2024-09-01T01:37:45.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-01T01:37:45.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-01T01:37:46.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-01T01:37:48.405 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-01T01:37:48.406 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:48.406 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:48.406 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:48.406 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:48.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:37:48.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-01T01:37:48.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:48.508 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:48.509 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:48.509 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:48.509 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:48.509 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:48.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-01T01:37:48.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:48.599 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:48.600 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:48.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-01T01:37:48.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-01T01:37:48.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-01T01:37:49.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-01T01:37:49.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-01T01:37:51.616 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-01T01:37:51.616 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:51.616 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:51.616 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:51.616 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:51.617 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:37:51.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:51.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:51.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-01T01:37:51.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:51.718 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:51.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:51.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:51.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:51.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-01T01:37:51.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:51.809 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:51.809 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:51.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:51.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:52.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:52.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-01T01:37:52.624 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already down. 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:52.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:37:52.637 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:37:52.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:52.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-01T01:37:53.070 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/510913172,v1:127.0.0.1:6803/510913172] [v2:127.0.0.1:6804/510913172,v1:127.0.0.1:6805/510913172] exists 24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:37:53.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:53.075 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:37:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2024-09-01T01:37:53.077 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:37:53.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T01:37:53.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-01T01:37:53.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2024-09-01T01:37:53.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T01:37:53.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T01:37:53.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T01:37:53.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T01:37:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:53.100+0000 7fc4e273e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:53.101+0000 7fc4e273e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:53.103 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:53.103+0000 7fc4e273e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:37:53.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:37:53.477 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:37:53.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:53.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:53.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:54.199 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:54.199+0000 7fc4e273e640 -1 Falling back to public interface 2024-09-01T01:37:54.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:54.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:54.888 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:37:54.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:37:54.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:54.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:55.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:56.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:56.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:56.357 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:37:56.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:37:56.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:56.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:56.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:37:57.000 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:37:56.999+0000 7fc4e273e640 -1 osd.0 6 log_to_monitors true 2024-09-01T01:37:57.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:37:57.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:37:57.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:37:57.758 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:37:57.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:37:57.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:37:58.191 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2523580797,v1:127.0.0.1:6803/2523580797] [v2:127.0.0.1:6804/2523580797,v1:127.0.0.1:6805/2523580797] exists,up 24053e71-2380-4e71-b9f8-cbfec990a294 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:58.192 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:58.193 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.193 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.193 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:58.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:58.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:37:58.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:58.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:58.280 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-01T01:37:58.280 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:58.280 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:58.281 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:58.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:58.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:37:58.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok dump_recovery_reservations 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:58.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:58.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:58.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:58.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:58.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:58.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:37:58.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:37:58.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:37:58.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:37:58.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:37:58.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:37:58.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:37:58.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:37:58.568 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:37:58.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:37:58.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:37:58.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:37:58.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:58.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:58.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:58.578 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:37:58.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:37:58.579 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:37:58.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:37:58.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:37:58.581 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:37:58.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:37:58.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:37:58.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:37:58.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:37:58.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:37:58.584 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:37:58.585 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.585 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:37:58.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:37:58.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:37:58.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:37:58.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:37:58.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:37:58.588 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:37:58.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:37:58.589 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:37:58.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:37:58.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:37:58.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:37:58.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:37:58.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:37:58.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:37:58.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:37:58.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:37:58.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:37:58.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:37:58.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:37:58.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:37:58.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:37:58.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:37:58.596 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:37:58.596 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:58.596 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:58.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-01T01:37:58.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:37:58.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:37:59.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:37:59.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:37:59.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:37:59.121 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:37:59.121 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:59.122 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:59.122 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:37:59.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:37:59.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:37:59.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:37:59.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:37:59.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:37:59.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:37:59.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:37:59.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:37:59.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:00.327 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:38:00.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:38:00.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:38:00.358 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:38:00.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:38:00.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:38:00.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:38:00.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:38:00.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bbd07028-695d-44cd-90db-d29c05202f8d 2024-09-01T01:38:00.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bbd07028-695d-44cd-90db-d29c05202f8d' 2024-09-01T01:38:00.361 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 bbd07028-695d-44cd-90db-d29c05202f8d 2024-09-01T01:38:00.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:38:00.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4xdNmvvtcFhAAyFrw5pfBfWiK/ZNpcQCo+w== 2024-09-01T01:38:00.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4xdNmvvtcFhAAyFrw5pfBfWiK/ZNpcQCo+w=="}' 2024-09-01T01:38:00.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bbd07028-695d-44cd-90db-d29c05202f8d -i td/mclock-config/0/new.json 2024-09-01T01:38:00.678 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:38:00.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:38:00.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQB4xdNmvvtcFhAAyFrw5pfBfWiK/ZNpcQCo+w== --osd-uuid bbd07028-695d-44cd-90db-d29c05202f8d 2024-09-01T01:38:00.717 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:00.716+0000 7f165f424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:00.747 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:00.746+0000 7f165f424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:00.748 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:00.747+0000 7f165f424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:00.748 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:00.747+0000 7f165f424640 -1 bdev(0x560d6e82bc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:38:00.748 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:00.747+0000 7f165f424640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:38:06.458 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:38:06.458 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:38:06.459 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:38:06.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:38:06.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:38:07.027 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:38:07.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:38:07.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:38:07.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:38:07.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:38:07.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:38:07.053 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:07.053+0000 7f821760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:07.055 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:07.055+0000 7f821760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:07.056 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:07.056+0000 7f821760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:07.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:38:07.476 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:38:07.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:07.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:07.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:08.653 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:08.653+0000 7f821760d640 -1 Falling back to public interface 2024-09-01T01:38:08.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:08.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:08.893 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:38:08.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:38:08.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:08.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:09.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:10.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:10.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:10.303 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:38:10.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:38:10.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:10.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:10.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:11.239 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:11.238+0000 7f821760d640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:38:11.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:11.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:11.729 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:38:11.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:38:11.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:11.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:12.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:13.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:13.268 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:38:13.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:13.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:38:13.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:13.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:13.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:14.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:14.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:14.677 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:38:14.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:38:14.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:14.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:15.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:16.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:16.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:16.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:38:16.104 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:38:16.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:16.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:16.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:17.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:17.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:17.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:38:17.527 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:38:17.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:17.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:17.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:18.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:18.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:18.945 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:38:18.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:38:18.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:18.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:19.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:20.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:20.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:20.363 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:38:20.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:38:20.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:20.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:20.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:21.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:21.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:21.773 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:38:21.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:38:21.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:21.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:22.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:23.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:23.550 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:38:23.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:23.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:38:23.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:23.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:23.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:24.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:24.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:24.981 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:38:24.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:38:24.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:24.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:25.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:26.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:26.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:26.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:38:26.680 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:38:26.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:26.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:27.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:28.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:28.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:28.793 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:38:28.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:38:28.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:28.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:29.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:30.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:30.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:30.266 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:38:30.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:38:30.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:30.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:30.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:31.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:31.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:31.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:38:31.680 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:38:31.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:31.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:32.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:33.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:33.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:33.367 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:38:33.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:38:33.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:33.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:33.849 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3113603791,v1:127.0.0.1:6803/3113603791] [v2:127.0.0.1:6804/3113603791,v1:127.0.0.1:6805/3113603791] exists,up bbd07028-695d-44cd-90db-d29c05202f8d 2024-09-01T01:38:33.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:38:33.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:38:33.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:38:33.851 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-01T01:38:34.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-01T01:38:34.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:38:34.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-01T01:38:34.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-01T01:38:34.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-01T01:38:34.545 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:38:34.545 INFO:tasks.workunit.client.0.smithi037.stdout: "success": "" 2024-09-01T01:38:34.545 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:34.556 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:38:34.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-01T01:38:34.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:38:34.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-01T01:38:34.642 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-01T01:38:34.642 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-01T01:38:34.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-01T01:38:34.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-01T01:38:34.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:34.645 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:34.646 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:38:34.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-01T01:38:34.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:38:34.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-01T01:38:34.730 INFO:tasks.workunit.client.0.smithi037.stdout:client_res = 0.300000 2024-09-01T01:38:34.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-01T01:38:34.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-01T01:38:34.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-01T01:38:34.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-01T01:38:34.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-01T01:38:34.732 INFO:tasks.workunit.client.0.smithi037.stdout:client_res_new = .400000 2024-09-01T01:38:34.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-01T01:38:35.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:38:35.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-01T01:38:35.662 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-01T01:38:35.662 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-01T01:38:35.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:38:35.664 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:35.665 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.665 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.665 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:38:35.665 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-01T01:38:35.665 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:38:35.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-01T01:38:35.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-01T01:38:35.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-01T01:38:35.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-01T01:38:35.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:38:35.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:38:35.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:38:35.858 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:38:35.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:38:35.859 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:38:35.861 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:38:35.861 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:38:35.861 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:38:35.861 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:38:35.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:38:35.862 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:38:35.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:38:35.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:38:35.868 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:38:35.869 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.869 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:38:35.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:38:35.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:38:35.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:38:35.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:38:35.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:38:35.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:38:35.872 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:38:35.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:38:35.873 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:38:35.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:38:35.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:38:35.874 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:38:35.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:38:35.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:38:35.875 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:38:35.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:38:35.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:38:35.878 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:38:35.878 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.878 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:38:35.880 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:38:35.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:38:35.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:38:35.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:38:35.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:38:35.881 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:38:35.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:38:35.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:38:35.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:38:35.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:38:35.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:38:35.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:38:35.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:38:35.885 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:38:35.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:38:35.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:38:35.887 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:38:35.887 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.888 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:38:35.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:38:35.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:38:35.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:38:35.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:38:35.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:38:35.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:38:35.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:38:35.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:38:35.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:38:35.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:38:35.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:38:35.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:38:35.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:38:35.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:38:35.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:38:35.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:38:35.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:38:35.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:38:35.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:38:35.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:38:35.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:38:35.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:38:35.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:38:35.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:38:35.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:38:35.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:38:35.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:38:35.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:38:35.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:38:35.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:35.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:35.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:38:35.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:38:35.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:36.331 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:36.332 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:38:36.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:38:36.516 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:38:36.516 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:38:36.516 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:38:36.516 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:38:36.517 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:38:36.517 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:38:36.517 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:38:36.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:38:36.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:38:36.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:38:36.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:38:36.606 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:38:36.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:38:37.540 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:38:37.540 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:38:37.540 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:38:37.540 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:38:37.541 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:37.541 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:37.541 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:37.541 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:38:37.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:38:37.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:38:37.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:38:37.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:38:37.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:38:37.571 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:38:37.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:38:37.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc96950a-233e-436d-9eca-e0b7d70b80bf 2024-09-01T01:38:37.574 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 dc96950a-233e-436d-9eca-e0b7d70b80bf 2024-09-01T01:38:37.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dc96950a-233e-436d-9eca-e0b7d70b80bf' 2024-09-01T01:38:37.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:38:37.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdxdNmJYkaIxAAsblIiAK4MzFovvaMW16BvQ== 2024-09-01T01:38:37.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdxdNmJYkaIxAAsblIiAK4MzFovvaMW16BvQ=="}' 2024-09-01T01:38:37.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc96950a-233e-436d-9eca-e0b7d70b80bf -i td/mclock-config/0/new.json 2024-09-01T01:38:37.864 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:38:37.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:38:37.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCdxdNmJYkaIxAAsblIiAK4MzFovvaMW16BvQ== --osd-uuid dc96950a-233e-436d-9eca-e0b7d70b80bf 2024-09-01T01:38:37.904 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:37.903+0000 7fe0ba610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:37.933 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:37.933+0000 7fe0ba610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:37.934 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:37.934+0000 7fe0ba610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:37.935 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:37.935+0000 7fe0ba610640 -1 bdev(0x55d37ed87c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:38:37.935 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:37.935+0000 7fe0ba610640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:38:43.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:38:43.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:38:43.716 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:38:43.716 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:38:43.716 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:38:44.297 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:38:44.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:38:44.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:38:44.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:38:44.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:38:44.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:38:44.323 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:44.323+0000 7f1ea14e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:44.325 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:44.325+0000 7f1ea14e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:44.326 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:44.326+0000 7f1ea14e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:38:44.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:44.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:38:44.744 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:38:44.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:44.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:45.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:45.416 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:45.416+0000 7f1ea14e0640 -1 Falling back to public interface 2024-09-01T01:38:46.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:46.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:46.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:38:46.187 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:38:46.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:46.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:46.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:47.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:47.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:47.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:38:47.630 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:38:47.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:47.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:48.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:48.433 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:38:48.432+0000 7f1ea14e0640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:38:49.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:49.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:49.039 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:38:49.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:38:49.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:49.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:49.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:38:50.499 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:38:50.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:50.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:50.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:51.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:51.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:51.909 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:38:51.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:38:51.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:51.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:52.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:53.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:53.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:53.317 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:38:53.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:38:53.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:53.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:53.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:54.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:54.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:54.738 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:38:54.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:38:54.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:54.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:55.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:56.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:56.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:56.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:38:56.187 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:38:56.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:56.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:56.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:57.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:57.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:57.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:38:57.617 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:38:57.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:57.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:58.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:38:59.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:38:59.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:38:59.027 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:38:59.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:38:59.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:38:59.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:38:59.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:00.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:00.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:00.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:39:00.696 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:39:00.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:00.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:01.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:02.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:02.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:02.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:39:02.608 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:39:02.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:02.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:03.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:04.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:04.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:04.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:39:04.461 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:39:04.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:04.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:05.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:06.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:06.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:06.312 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:39:06.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:39:06.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:06.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:07.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:08.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:08.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:08.218 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:39:08.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:39:08.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:08.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:09.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:10.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:10.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:10.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:39:10.278 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:39:10.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:10.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:10.691 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1272754207,v1:127.0.0.1:6803/1272754207] [v2:127.0.0.1:6804/1272754207,v1:127.0.0.1:6805/1272754207] exists,up dc96950a-233e-436d-9eca-e0b7d70b80bf 2024-09-01T01:39:10.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:39:10.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:39:10.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:39:10.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-01T01:39:10.692 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-01T01:39:11.151 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-01T01:39:11.151 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:11.152 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:11.153 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:11.153 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:11.153 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:11.153 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:39:11.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-01T01:39:11.238 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-01T01:39:11.239 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-01T01:39:11.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-01T01:39:11.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-01T01:39:11.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-01T01:39:11.342 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:39:11.343 INFO:tasks.workunit.client.0.smithi037.stdout: "success": "" 2024-09-01T01:39:11.343 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:11.354 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:11.355 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:11.355 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:11.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:11.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:11.442 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:11.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:39:11.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-01T01:39:11.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-01T01:39:11.527 INFO:tasks.workunit.client.0.smithi037.stdout:Original client_res for osd.0 = 0.300000 2024-09-01T01:39:11.527 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-01T01:39:11.528 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:11.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-01T01:39:11.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-01T01:39:11.528 INFO:tasks.workunit.client.0.smithi037.stdout:client_res_new = .400000 2024-09-01T01:39:11.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-01T01:39:11.992 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:39:12.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-01T01:39:12.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-01T01:39:12.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:12.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:12.431 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:12.432 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:12.432 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:12.432 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:12.432 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:12.432 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:39:12.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-01T01:39:12.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-01T01:39:12.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:12.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:12.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-01T01:39:12.624 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:39:12.624 INFO:tasks.workunit.client.0.smithi037.stdout: "success": "" 2024-09-01T01:39:12.624 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:12.635 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:12.636 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:12.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:12.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:39:12.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-01T01:39:12.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-01T01:39:12.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-01T01:39:12.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:39:13.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-01T01:39:13.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-01T01:39:13.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:13.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:13.166 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:13.167 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:13.167 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:13.167 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:39:13.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-01T01:39:13.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-01T01:39:13.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:13.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:13.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-01T01:39:13.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-01T01:39:18.708 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:39:19.151 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-01T01:39:19.168 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-01T01:39:19.168 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-01T01:39:19.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:39:19.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-01T01:39:19.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-01T01:39:19.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:39:19.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:39:19.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:39:19.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:39:19.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:39:19.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:39:19.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:39:19.348 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:39:19.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:39:19.349 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:39:19.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:39:19.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:39:19.350 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:39:19.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:39:19.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:39:19.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:39:19.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:39:19.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:39:19.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:39:19.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:39:19.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:39:19.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:39:19.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:39:19.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:39:19.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:39:19.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:39:19.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:39:19.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:39:19.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:39:19.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:39:19.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:39:19.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:39:19.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:39:19.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:39:19.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:39:19.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:39:19.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:39:19.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:39:19.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:39:19.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:39:19.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:39:19.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:39:19.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:39:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:39:19.376 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:39:19.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:39:19.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:39:19.377 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:39:19.377 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.377 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:39:19.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:39:19.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:39:19.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:39:19.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:39:19.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:39:19.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:39:19.382 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:39:19.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:39:19.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:39:19.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:39:19.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:39:19.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:39:19.384 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:39:19.384 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:39:19.385 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:39:19.385 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:39:19.385 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:39:19.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:39:19.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:39:19.386 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:39:19.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:39:19.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:39:19.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:39:19.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:39:19.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:39:19.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:39:19.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:39:19.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:39:19.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:39:19.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:39:19.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:39:19.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:39:19.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:39:19.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:19.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:19.830 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:39:19.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:39:20.013 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:39:20.014 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:39:20.014 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:39:20.014 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:39:20.014 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:20.015 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:39:20.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:39:20.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:39:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:39:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:39:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:39:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:39:20.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:39:20.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:21.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:39:21.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:39:21.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:39:21.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:39:21.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:39:21.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:39:21.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:39:21.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e 2024-09-01T01:39:21.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e' 2024-09-01T01:39:21.063 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e 2024-09-01T01:39:21.063 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:39:21.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJxdNm/tSfBBAAxBR6d+EYZhfS+5SmoVymww== 2024-09-01T01:39:21.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJxdNm/tSfBBAAxBR6d+EYZhfS+5SmoVymww=="}' 2024-09-01T01:39:21.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e -i td/mclock-config/0/new.json 2024-09-01T01:39:21.348 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:39:21.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:39:21.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDJxdNm/tSfBBAAxBR6d+EYZhfS+5SmoVymww== --osd-uuid 5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e 2024-09-01T01:39:21.387 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:21.387+0000 7efc2760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:21.415 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:21.415+0000 7efc2760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:21.416 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:21.416+0000 7efc2760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:21.416 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:21.416+0000 7efc2760e640 -1 bdev(0x562bff58fc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:39:21.417 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:21.416+0000 7efc2760e640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:39:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:39:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:39:26.971 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:39:26.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:39:26.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:39:27.648 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:39:27.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:39:27.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:39:27.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:39:27.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:39:27.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:39:27.675 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:27.674+0000 7f1aa6644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:27.676 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:27.676+0000 7f1aa6644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:27.677 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:27.677+0000 7f1aa6644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:39:28.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:39:28.084 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:39:28.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:28.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:28.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:29.016 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:29.016+0000 7f1aa6644640 -1 Falling back to public interface 2024-09-01T01:39:29.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:29.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:29.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:39:29.519 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:39:29.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:29.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:29.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:30.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:30.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:30.954 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:39:30.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:39:30.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:30.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:31.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:32.131 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:39:32.131+0000 7f1aa6644640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:39:32.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:32.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:32.395 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:39:32.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:39:32.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:32.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:32.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:33.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:33.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:33.876 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:39:33.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:39:33.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:33.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:34.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:35.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:35.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:35.302 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:39:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:39:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:35.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:36.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:36.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:36.714 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:39:36.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:39:36.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:36.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:37.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:38.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:38.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:38.171 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:39:38.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:39:38.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:38.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:38.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:39.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:39.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:39.598 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:39:39.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:39:39.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:39.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:40.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:41.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:41.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:41.038 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:39:41.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:39:41.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:41.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:41.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:42.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:42.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:42.469 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:39:42.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:39:42.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:42.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:42.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:43.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:43.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:43.985 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:39:43.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:39:43.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:43.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:44.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:45.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:45.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:45.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:39:45.405 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:39:45.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:45.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:46.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:47.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:47.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:47.577 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:39:47.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:39:47.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:47.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:48.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:49.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:49.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:49.304 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:39:49.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:39:49.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:49.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:50.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:51.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:51.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:51.119 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:39:51.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:39:51.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:51.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:51.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:52.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:52.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:52.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:39:52.804 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:39:52.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:52.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:53.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:39:54.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:39:54.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:39:54.531 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:39:54.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:39:54.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:39:54.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:39:54.953 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3932969370,v1:127.0.0.1:6803/3932969370] [v2:127.0.0.1:6804/3932969370,v1:127.0.0.1:6805/3932969370] exists,up 5c54e8a0-2f8b-4647-b99c-c7c6c3ced15e 2024-09-01T01:39:54.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:39:54.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:39:54.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:39:54.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-01T01:39:55.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-01T01:39:55.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-01T01:39:55.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-01T01:39:55.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-01T01:39:55.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:55.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:55.460 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:55.461 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:55.461 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:55.461 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:55.461 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-01T01:39:55.461 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:39:55.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-01T01:39:55.548 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-01T01:39:55.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-01T01:39:55.550 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:55.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-01T01:39:55.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-01T01:39:55.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:55.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-01T01:39:55.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-01T01:39:56.211 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-01T01:39:56.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-01T01:39:56.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-01T01:39:56.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-01T01:39:56.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-01T01:39:58.233 INFO:tasks.workunit.client.0.smithi037.stdout:Check configuration values - Attempt#: 0 2024-09-01T01:39:58.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-01T01:39:58.234 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:58.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-01T01:39:58.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-01T01:39:58.681 INFO:tasks.workunit.client.0.smithi037.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-01T01:39:58.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-01T01:39:58.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:58.682 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-01T01:39:58.682 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-01T01:39:58.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:58.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-01T01:39:58.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:39:59.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-01T01:39:59.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-01T01:39:59.082 INFO:tasks.workunit.client.0.smithi037.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-01T01:39:59.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-01T01:39:59.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:59.083 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-01T01:39:59.083 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:59.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:59.085 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:59.086 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:59.086 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:59.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-01T01:39:59.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-01T01:39:59.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-01T01:39:59.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-01T01:39:59.172 INFO:tasks.workunit.client.0.smithi037.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-01T01:39:59.172 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-01T01:39:59.172 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:59.173 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-01T01:39:59.173 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:59.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-01T01:39:59.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-01T01:39:59.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-01T01:39:59.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:39:59.176 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:39:59.177 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:39:59.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-01T01:39:59.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:39:59.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-01T01:39:59.262 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-01T01:39:59.263 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:39:59.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-01T01:39:59.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-01T01:39:59.858 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-01T01:39:59.859 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-01T01:39:59.860 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-01T01:39:59.860 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-01T01:39:59.860 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-01T01:40:01.861 INFO:tasks.workunit.client.0.smithi037.stdout:Check configuration values - Attempt#: 0 2024-09-01T01:40:01.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-01T01:40:01.862 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:40:02.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-01T01:40:02.274 INFO:tasks.workunit.client.0.smithi037.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-01T01:40:02.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-01T01:40:02.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-01T01:40:02.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:40:02.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-01T01:40:02.275 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-01T01:40:02.276 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-01T01:40:02.276 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-01T01:40:02.276 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:40:02.682 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-01T01:40:02.682 INFO:tasks.workunit.client.0.smithi037.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-01T01:40:02.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-01T01:40:02.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-01T01:40:02.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:40:02.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-01T01:40:02.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:40:02.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-01T01:40:02.686 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-01T01:40:02.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:40:02.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-01T01:40:02.773 INFO:tasks.workunit.client.0.smithi037.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-01T01:40:02.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-01T01:40:02.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-01T01:40:02.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:40:02.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-01T01:40:02.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:02.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:02.777 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:02.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:02.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:02.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:02.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:02.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:02.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:02.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:02.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:02.884 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:02.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:02.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:02.886 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:02.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:02.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:02.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:02.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:02.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:02.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:02.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:40:02.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:02.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:02.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:02.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:02.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:02.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:02.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:02.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:02.898 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:02.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:02.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:02.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:02.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:02.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:02.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:02.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.903 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:02.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:02.907 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:02.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:02.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:02.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:02.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:02.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:02.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:40:02.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:40:02.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:02.919 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:02.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:02.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:02.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:02.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:02.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:02.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:02.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:02.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:02.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:40:02.926 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:40:02.926 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:02.926 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:02.926 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:40:02.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:40:02.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:40:02.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:03.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:03.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:40:03.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:40:03.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:40:03.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:40:03.575 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:40:03.576 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:03.576 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:03.576 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:03.576 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:40:03.577 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:40:03.577 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:40:03.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:40:03.659 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:40:03.660 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:40:03.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:40:04.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:40:04.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:04.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:40:04.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:40:04.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:40:04.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:40:04.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:40:04.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:40:04.641 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:40:04.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:40:04.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:40:04.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:40:04.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:40:04.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:40:04.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2fb0a588-c179-491b-9924-82bfb3c1b7fe 2024-09-01T01:40:04.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2fb0a588-c179-491b-9924-82bfb3c1b7fe' 2024-09-01T01:40:04.645 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 2fb0a588-c179-491b-9924-82bfb3c1b7fe 2024-09-01T01:40:04.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:40:04.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0xdNmlYI5JxAA9yWgnrEOgokaLytSN083GQ== 2024-09-01T01:40:04.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0xdNmlYI5JxAA9yWgnrEOgokaLytSN083GQ=="}' 2024-09-01T01:40:04.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2fb0a588-c179-491b-9924-82bfb3c1b7fe -i td/mclock-config/0/new.json 2024-09-01T01:40:04.935 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:40:04.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:40:04.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQD0xdNmlYI5JxAA9yWgnrEOgokaLytSN083GQ== --osd-uuid 2fb0a588-c179-491b-9924-82bfb3c1b7fe 2024-09-01T01:40:04.974 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:04.973+0000 7f172880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:05.002 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:05.002+0000 7f172880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:05.003 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:05.003+0000 7f172880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:05.004 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:05.004+0000 7f172880f640 -1 bdev(0x55ab42ca5c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:40:05.004 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:05.004+0000 7f172880f640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:40:10.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:40:10.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:40:10.627 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:40:10.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:40:10.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:40:11.227 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:40:11.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:40:11.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:40:11.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:40:11.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:40:11.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:40:11.253 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:11.253+0000 7f24ca81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:11.257 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:11.256+0000 7f24ca81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:11.257 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:11.257+0000 7f24ca81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:11.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:40:11.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:40:11.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:40:11.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:40:11.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:40:11.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:11.679 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:40:11.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:40:11.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:11.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:12.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:12.355 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:12.355+0000 7f24ca81d640 -1 Falling back to public interface 2024-09-01T01:40:13.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:13.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:13.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:40:13.105 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:40:13.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:13.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:13.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:14.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:14.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:14.540 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:40:14.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:40:14.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:14.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:14.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:15.117 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:15.116+0000 7f24ca81d640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:40:15.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:15.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:15.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:40:15.944 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:40:15.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:15.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:16.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:17.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:17.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:40:17.390 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:40:17.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:17.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:17.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:18.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:18.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:18.915 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:40:18.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:40:18.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:18.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:19.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:20.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:20.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:20.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:40:20.346 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:40:20.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:20.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:20.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:21.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:21.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:21.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:40:21.948 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:40:21.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:21.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:22.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:23.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:23.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:23.373 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:40:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:40:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:23.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:24.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:24.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:24.896 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:40:24.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:40:24.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:24.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:25.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:26.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:26.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:26.301 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:40:26.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:40:26.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:26.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:26.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:27.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:27.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:27.864 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:40:27.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:40:27.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:27.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:28.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:29.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:29.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:29.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:40:29.487 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:40:29.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:29.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:30.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:31.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:31.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:31.193 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:40:31.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:40:31.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:31.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:33.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:33.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:33.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:40:33.215 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:40:33.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:33.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:34.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:35.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:35.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:35.335 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:40:35.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:40:35.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:35.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:36.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:37.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:37.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:37.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:40:37.116 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:40:37.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:37.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:37.533 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3856172853,v1:127.0.0.1:6803/3856172853] [v2:127.0.0.1:6804/3856172853,v1:127.0.0.1:6805/3856172853] exists,up 2fb0a588-c179-491b-9924-82bfb3c1b7fe 2024-09-01T01:40:37.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:40:37.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:40:37.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:40:37.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-01T01:40:37.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-01T01:40:37.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-01T01:40:37.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_profile 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-01T01:40:38.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-01T01:40:38.058 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:38.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-01T01:40:38.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-01T01:40:38.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:38.144 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-01T01:40:38.144 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-01T01:40:38.249 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:40:38.249 INFO:tasks.workunit.client.0.smithi037.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-01T01:40:38.249 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:40:38.259 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-01T01:40:38.260 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-01T01:40:38.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-01T01:40:38.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-01T01:40:38.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-01T01:40:40.262 INFO:tasks.workunit.client.0.smithi037.stdout:Check configuration values - Attempt#: 0 2024-09-01T01:40:40.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-01T01:40:40.263 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:40.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-01T01:40:40.696 INFO:tasks.workunit.client.0.smithi037.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-01T01:40:40.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-01T01:40:40.696 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-01T01:40:40.696 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:40.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-01T01:40:40.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-01T01:40:40.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:40.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-01T01:40:40.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:41.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-01T01:40:41.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-01T01:40:41.093 INFO:tasks.workunit.client.0.smithi037.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-01T01:40:41.094 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-01T01:40:41.094 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:41.095 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-01T01:40:41.095 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:41.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-01T01:40:41.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:41.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:41.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-01T01:40:41.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-01T01:40:41.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-01T01:40:41.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-01T01:40:41.183 INFO:tasks.workunit.client.0.smithi037.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-01T01:40:41.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-01T01:40:41.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-01T01:40:41.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:41.185 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-01T01:40:41.185 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:41.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-01T01:40:41.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-01T01:40:41.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-01T01:40:41.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-01T01:40:41.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:40:41.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-01T01:40:41.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-01T01:40:41.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:41.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-01T01:40:41.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-01T01:40:41.380 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:40:41.380 INFO:tasks.workunit.client.0.smithi037.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-01T01:40:41.380 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:40:41.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-01T01:40:41.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-01T01:40:41.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-01T01:40:41.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-01T01:40:41.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-01T01:40:43.393 INFO:tasks.workunit.client.0.smithi037.stdout:Check configuration values - Attempt#: 0 2024-09-01T01:40:43.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-01T01:40:43.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-01T01:40:43.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-01T01:40:43.814 INFO:tasks.workunit.client.0.smithi037.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-01T01:40:43.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-01T01:40:43.815 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-01T01:40:43.815 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:43.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-01T01:40:43.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-01T01:40:43.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-01T01:40:43.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-01T01:40:43.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:44.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-01T01:40:44.218 INFO:tasks.workunit.client.0.smithi037.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-01T01:40:44.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-01T01:40:44.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-01T01:40:44.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:44.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-01T01:40:44.220 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:44.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-01T01:40:44.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-01T01:40:44.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-01T01:40:44.221 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-01T01:40:44.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:40:44.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:40:44.222 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:44.222 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.223 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.223 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:40:44.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-01T01:40:44.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-01T01:40:44.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-01T01:40:44.309 INFO:tasks.workunit.client.0.smithi037.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-01T01:40:44.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-01T01:40:44.310 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-01T01:40:44.310 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:44.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-01T01:40:44.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:44.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:44.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:44.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:44.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:44.420 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:44.421 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:44.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:44.422 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:44.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:44.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:44.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:44.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:44.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:44.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:44.432 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:44.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:44.433 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:44.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:44.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:44.435 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:44.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:44.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:44.437 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:44.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:44.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:44.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:44.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:44.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:44.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:44.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-01T01:40:44.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-01T01:40:44.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:44.445 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:44.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:44.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:44.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:44.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:44.451 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:44.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:44.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:44.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:40:44.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:40:44.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.454 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-01T01:40:44.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:40:44.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:40:44.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:40:44.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:40:44.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2024-09-01T01:40:44.465 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16001 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2024-09-01T01:40:44.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-01T01:40:44.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:40:44.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:44.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:40:45.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:40:45.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:40:45.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:40:45.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:40:45.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:40:45.129 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:40:45.130 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get fsid 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:45.218 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:45.219 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-mon.a.asok 2024-09-01T01:40:45.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:40:45.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16001/ceph-mon.a.asok config get mon_host 2024-09-01T01:40:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-01T01:40:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2024-09-01T01:40:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:40:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:40:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:40:45.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2024-09-01T01:40:45.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:40:46.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:40:46.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:46.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:46.144 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:46.145 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:46.145 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:46.145 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:46.145 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:40:46.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-01T01:40:46.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-01T01:40:46.175 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:40:46.176 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:40:46.176 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:40:46.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:40:46.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:40:46.176 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:40:46.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:40:46.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T01:40:46.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2024-09-01T01:40:46.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:40:46.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a822618-6a80-49bc-af79-53164c2b3576 2024-09-01T01:40:46.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a822618-6a80-49bc-af79-53164c2b3576' 2024-09-01T01:40:46.179 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 3a822618-6a80-49bc-af79-53164c2b3576 2024-09-01T01:40:46.179 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:40:46.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAextNm2m5wCxAASF0ZbTLqqxVvCOUK0qHZtA== 2024-09-01T01:40:46.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAextNm2m5wCxAASF0ZbTLqqxVvCOUK0qHZtA=="}' 2024-09-01T01:40:46.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a822618-6a80-49bc-af79-53164c2b3576 -i td/mclock-config/0/new.json 2024-09-01T01:40:46.485 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:40:46.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2024-09-01T01:40:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAextNm2m5wCxAASF0ZbTLqqxVvCOUK0qHZtA== --osd-uuid 3a822618-6a80-49bc-af79-53164c2b3576 2024-09-01T01:40:46.524 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:46.524+0000 7f01f1618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:46.554 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:46.554+0000 7f01f1618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:46.555 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:46.555+0000 7f01f1618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:46.555 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:46.555+0000 7f01f1618640 -1 bdev(0x5604b9f1dc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:40:46.556 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:46.555+0000 7f01f1618640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-01T01:40:52.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-01T01:40:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:40:52.495 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:40:52.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:40:52.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:40:53.059 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:40:53.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:40:53.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=7b8953e0-0af7-4f1e-9266-d4f77cee80ea --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.16001/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T01:40:53.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:40:53.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:40:53.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:40:53.086 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:53.085+0000 7f97f1819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:53.088 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:53.088+0000 7f97f1819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:53.089 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:53.089+0000 7f97f1819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:53.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:40:53.506 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:40:53.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:53.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:53.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:54.424 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:54.424+0000 7f97f1819640 -1 Falling back to public interface 2024-09-01T01:40:54.940 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:54.940 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:40:54.941 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:40:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:55.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:56.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:56.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:56.376 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:40:56.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:40:56.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:56.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:56.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:57.387 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:40:57.386+0000 7f97f1819640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:40:57.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:57.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:57.801 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:40:57.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:40:57.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:57.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:58.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:40:59.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:40:59.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:40:59.408 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:40:59.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:40:59.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:40:59.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:40:59.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:00.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:00.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:00.856 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:41:00.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:41:00.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:00.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:01.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:02.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:02.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:02.283 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:41:02.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:41:02.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:02.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:02.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:03.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:03.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:03.701 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:41:03.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:41:03.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:03.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:04.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:05.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:05.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:05.216 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:41:05.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:41:05.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:05.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:05.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:06.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:06.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:06.641 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:41:06.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:41:06.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:06.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:07.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:08.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:08.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:08.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:41:08.104 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:41:08.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:08.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:08.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:09.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:09.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:09.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:41:09.692 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:41:09.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:09.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:10.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:11.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:11.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:11.550 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:41:11.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:41:11.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:11.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:12.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:13.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:13.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:13.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:41:13.200 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:41:13.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:13.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:14.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:15.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:15.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:15.034 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:41:15.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:41:15.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:15.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:16.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:17.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:17.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:17.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:41:17.122 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:41:17.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:17.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:18.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:19.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:19.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:19.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:41:19.248 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:41:19.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:19.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:19.692 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2260298540,v1:127.0.0.1:6803/2260298540] [v2:127.0.0.1:6804/2260298540,v1:127.0.0.1:6805/2260298540] exists,up 3a822618-6a80-49bc-af79-53164c2b3576 2024-09-01T01:41:19.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:41:19.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:41:19.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:41:19.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-01T01:41:19.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-01T01:41:19.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-01T01:41:19.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-01T01:41:19.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-01T01:41:20.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-01T01:41:22.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:41:22.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-01T01:41:22.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-01T01:41:22.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-01T01:41:22.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-01T01:41:22.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-01T01:41:23.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-01T01:41:25.407 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16001/ceph-osd.0.asok 2024-09-01T01:41:25.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-01T01:41:25.409 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.16001/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:41:25.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:41:25.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:41:25.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:41:25.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:41:25.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:41:25.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:41:25.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:41:25.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:41:25.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:41:25.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:41:25.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:41:25.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:41:25.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:41:25.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:41:25.622 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:41:25.622 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.622 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:25.622 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:41:25.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:41:25.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:41:25.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:41:25.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:41:25.625 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:41:25.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:41:25.627 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:41:25.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:41:25.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:41:25.628 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:41:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:41:25.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:41:25.629 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:41:25.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:41:25.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:41:25.632 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:41:25.632 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.632 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:25.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:41:25.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:41:25.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:41:25.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/mclock-config 0 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:41:25.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:41:25.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:41:25.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:41:25.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:41:25.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:41:25.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:41:25.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:41:25.637 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:41:25.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:41:25.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:41:25.639 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:41:25.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T01:41:25.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2024-09-01T01:41:25.641 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:41:25.642 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.642 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16001 2024-09-01T01:41:25.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16001 2024-09-01T01:41:25.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:41:25.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:41:25.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T01:41:25.644 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T01:41:25.644 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T01:41:25.670 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-01T01:41:25.670 DEBUG:teuthology.orchestra.run.smithi037:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-01T01:41:25.731 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T01:41:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T01:41:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:41:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:41:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/network-ping 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none ' 2024-09-01T01:41:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-01T01:41:25.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-01T01:41:25.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-01T01:41:25.739 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_network_ping_test2' 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-01T01:41:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:41:25.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:41:25.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:41:25.744 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:41:25.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:41:25.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:41:25.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:41:25.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:41:25.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:41:25.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-01T01:41:25.747 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:41:25.747 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.747 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:25.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30030 2024-09-01T01:41:25.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:41:25.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:41:25.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2024-09-01T01:41:25.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:41:25.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:25.750 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:25.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.30030 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:41:25.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2024-09-01T01:41:25.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:26.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:41:26.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:41:26.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:41:26.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:41:26.399 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mon.a.asok 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:41:26.400 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30030/ceph-mon.a.asok config get fsid 2024-09-01T01:41:26.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:41:26.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:41:26.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:41:26.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:41:26.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mon.a.asok 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:41:26.487 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30030/ceph-mon.a.asok config get mon_host 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2024-09-01T01:41:26.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:41:27.421 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:41:27.454 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:41:27.458 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-01T01:41:27.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:41:27.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=02e5be1c-cc74-4b6e-a40b-6e682daa87c1 2024-09-01T01:41:27.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 02e5be1c-cc74-4b6e-a40b-6e682daa87c1' 2024-09-01T01:41:27.462 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 02e5be1c-cc74-4b6e-a40b-6e682daa87c1 2024-09-01T01:41:27.462 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:41:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHxtNm5XYnHBAAYfV/ukf9Dnlri0HmFokYHg== 2024-09-01T01:41:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHxtNm5XYnHBAAYfV/ukf9Dnlri0HmFokYHg=="}' 2024-09-01T01:41:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 02e5be1c-cc74-4b6e-a40b-6e682daa87c1 -i td/network-ping/0/new.json 2024-09-01T01:41:27.738 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:41:27.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2024-09-01T01:41:27.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHxtNm5XYnHBAAYfV/ukf9Dnlri0HmFokYHg== --osd-uuid 02e5be1c-cc74-4b6e-a40b-6e682daa87c1 2024-09-01T01:41:27.780 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:27.778+0000 7f45d120d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:27.807 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:27.806+0000 7f45d120d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:27.808 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:27.807+0000 7f45d120d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:27.808 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:27.807+0000 7f45d120d640 -1 bdev(0x5611c3d55c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:41:27.808 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:27.807+0000 7f45d120d640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-01T01:41:33.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-01T01:41:33.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:41:33.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:41:33.246 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:41:33.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:41:33.816 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:41:33.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:41:33.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:41:33.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:41:33.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:41:33.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:41:33.844 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:33.844+0000 7fc8fb6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:33.846 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:33.846+0000 7fc8fb6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:33.847 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:33.847+0000 7fc8fb6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:41:34.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:41:34.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:41:34.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:41:34.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:41:34.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:41:34.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:34.252 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:41:34.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:41:34.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:34.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:34.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:35.204 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:35.204+0000 7fc8fb6e2640 -1 Falling back to public interface 2024-09-01T01:41:35.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:35.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:35.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:41:35.651 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:41:35.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:35.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:36.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:37.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:37.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:37.087 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:41:37.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:41:37.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:37.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:37.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:38.033 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:41:38.032+0000 7fc8fb6e2640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:41:38.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:38.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:38.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:41:38.511 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:41:38.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:38.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:39.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:40.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:40.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:40.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:41:40.111 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:41:40.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:40.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:40.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:41.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:41.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:41.525 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:41:41.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:41:41.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:41.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:41.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:42.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:42.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:42.956 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:41:42.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:41:42.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:42.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:43.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:44.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:44.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:44.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:41:44.379 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:41:44.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:44.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:44.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:45.917 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:45.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:45.918 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:41:45.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:41:45.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:45.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:46.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:47.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:47.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:47.348 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:41:47.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:41:47.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:47.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:47.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:48.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:48.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:48.828 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:41:48.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:41:48.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:48.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:49.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:50.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:50.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:50.237 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:41:50.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:41:50.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:50.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:50.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:51.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:51.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:51.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:41:51.732 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:41:51.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:51.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:52.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:53.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:53.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:53.685 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:41:53.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:41:53.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:53.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:54.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:55.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:55.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:55.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:41:55.475 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:41:55.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:55.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:56.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:57.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:57.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:57.123 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:41:57.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:41:57.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:57.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:57.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:41:58.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:41:58.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:41:58.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:41:58.786 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:41:58.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:41:58.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:41:59.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:00.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:00.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:00.421 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:42:00.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:42:00.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:00.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:42:00.866 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3643148675,v1:127.0.0.1:6803/3643148675] [v2:127.0.0.1:6804/3643148675,v1:127.0.0.1:6805/3643148675] exists,up 02e5be1c-cc74-4b6e-a40b-6e682daa87c1 2024-09-01T01:42:00.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:42:00.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:42:00.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:42:00.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:42:00.871 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:42:00.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:42:00.873 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 e858bbbc-478c-449f-803c-6b3741efda27 2024-09-01T01:42:00.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e858bbbc-478c-449f-803c-6b3741efda27 2024-09-01T01:42:00.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e858bbbc-478c-449f-803c-6b3741efda27' 2024-09-01T01:42:00.874 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:42:00.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoxtNmVtffNBAA3SVt7IPnY3b3m2TqneiSjw== 2024-09-01T01:42:00.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoxtNmVtffNBAA3SVt7IPnY3b3m2TqneiSjw=="}' 2024-09-01T01:42:00.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e858bbbc-478c-449f-803c-6b3741efda27 -i td/network-ping/1/new.json 2024-09-01T01:42:01.506 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:42:01.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2024-09-01T01:42:01.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoxtNmVtffNBAA3SVt7IPnY3b3m2TqneiSjw== --osd-uuid e858bbbc-478c-449f-803c-6b3741efda27 2024-09-01T01:42:01.548 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:01.547+0000 7fafa83c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:01.549 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:01.549+0000 7fafa83c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:01.550 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:01.550+0000 7fafa83c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:01.551 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:01.551+0000 7fafa83c5640 -1 bdev(0x560b1d36fc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-01T01:42:01.551 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:01.551+0000 7fafa83c5640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-01T01:42:07.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-01T01:42:07.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:42:07.597 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T01:42:07.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T01:42:07.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:42:08.158 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T01:42:08.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T01:42:08.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:42:08.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:42:08.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:42:08.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:42:08.186 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:08.185+0000 7f7fbe419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:08.192 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:08.192+0000 7f7fbe419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:08.193 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:08.193+0000 7f7fbe419640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:42:08.615 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:42:08.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:08.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:09.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:09.527 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:09.527+0000 7f7fbe419640 -1 Falling back to public interface 2024-09-01T01:42:10.050 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:10.050 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:10.050 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:42:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:42:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:10.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:11.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:11.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:11.485 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:42:11.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:42:11.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:11.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:11.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:12.504 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:12.504+0000 7f7fbe419640 -1 osd.1 0 log_to_monitors true 2024-09-01T01:42:12.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:12.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:12.953 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:42:12.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:42:12.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:12.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:13.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:14.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:14.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:14.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:42:14.398 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:42:14.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:14.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:14.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:15.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:15.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:15.872 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:42:15.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:42:15.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:15.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:16.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:17.422 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:42:17.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:42:17.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:17.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:17.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:18.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:18.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:18.850 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:42:18.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:42:18.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:18.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:19.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:20.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:20.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:20.414 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:42:20.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:42:20.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:20.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:20.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:21.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:21.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:21.912 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:42:21.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:42:21.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:21.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:22.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:23.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:23.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:23.415 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:42:23.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:42:23.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:23.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:23.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:24.846 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:24.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:24.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:42:24.847 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:42:24.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:24.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:25.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:26.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:26.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:26.342 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:42:26.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:42:26.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:26.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:27.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:28.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:28.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:28.436 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:42:28.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:42:28.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:28.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:28.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:29.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:29.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:29.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:42:29.933 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:42:29.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:29.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:30.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:31.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:31.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:31.950 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:42:31.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:42:31.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:31.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:32.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:33.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:33.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:33.791 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:42:33.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:42:33.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:33.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:34.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:35.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:35.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:35.801 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:42:35.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:42:35.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:35.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:42:36.269 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1726333237,v1:127.0.0.1:6811/1726333237] [v2:127.0.0.1:6812/1726333237,v1:127.0.0.1:6813/1726333237] exists,up e858bbbc-478c-449f-803c-6b3741efda27 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T01:42:36.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:42:36.274 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:42:36.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ae624dbf-adf1-494c-84f1-2adf091ea1b5 2024-09-01T01:42:36.276 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 ae624dbf-adf1-494c-84f1-2adf091ea1b5 2024-09-01T01:42:36.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ae624dbf-adf1-494c-84f1-2adf091ea1b5' 2024-09-01T01:42:36.277 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:42:36.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMxtNmSANoERAATUSLVZvtiZ2Zb4onMOVs9g== 2024-09-01T01:42:36.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMxtNmSANoERAATUSLVZvtiZ2Zb4onMOVs9g=="}' 2024-09-01T01:42:36.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ae624dbf-adf1-494c-84f1-2adf091ea1b5 -i td/network-ping/2/new.json 2024-09-01T01:42:36.882 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:42:36.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2024-09-01T01:42:36.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMxtNmSANoERAATUSLVZvtiZ2Zb4onMOVs9g== --osd-uuid ae624dbf-adf1-494c-84f1-2adf091ea1b5 2024-09-01T01:42:36.925 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:36.923+0000 7f8b07b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:36.926 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:36.925+0000 7f8b07b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:36.927 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:36.926+0000 7f8b07b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:36.928 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:36.927+0000 7f8b07b4b640 -1 bdev(0x55fa4d1efc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-01T01:42:36.928 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:36.927+0000 7f8b07b4b640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-01T01:42:43.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-01T01:42:43.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:42:43.237 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-01T01:42:43.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T01:42:43.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:42:43.797 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-01T01:42:43.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T01:42:43.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:42:43.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:42:43.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:42:43.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:42:43.825 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:43.825+0000 7f3f2acf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:43.828 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:43.828+0000 7f3f2acf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:43.829 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:43.829+0000 7f3f2acf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:42:44.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:44.251 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:42:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:42:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:44.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:45.441 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:45.441+0000 7f3f2acf1640 -1 Falling back to public interface 2024-09-01T01:42:45.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:45.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:45.663 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:42:45.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:42:45.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:45.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:46.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:47.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:47.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:47.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:42:47.104 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:42:47.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:47.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:47.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:48.158 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:42:48.158+0000 7f3f2acf1640 -1 osd.2 0 log_to_monitors true 2024-09-01T01:42:48.539 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:48.539 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:42:48.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:48.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:42:48.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:48.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:48.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:49.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:49.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:49.979 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:42:49.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:42:49.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:49.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:50.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:51.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:51.428 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:42:51.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:51.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:42:51.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:51.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:51.987 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:52.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:52.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:52.989 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:42:52.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:42:52.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:52.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:53.453 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:54.455 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:54.455 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:42:54.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:54.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:42:54.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:54.456 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:54.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:55.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:55.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:55.945 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:42:55.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:42:55.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:55.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:56.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:57.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:57.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:57.377 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:42:57.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:42:57.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:57.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:57.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:42:58.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:42:58.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:42:58.817 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:42:58.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:42:58.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:42:58.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:42:59.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:00.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:00.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:00.314 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:43:00.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:43:00.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:00.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:00.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:01.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:01.890 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:43:01.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:01.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:43:01.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:01.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:02.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:03.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:03.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:03.908 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:43:03.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:43:03.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:03.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:04.600 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:05.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:05.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:05.602 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:43:05.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:43:05.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:05.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:06.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:07.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:07.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:07.352 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:43:07.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:43:07.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:07.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:08.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:09.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:09.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:09.069 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:43:09.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:43:09.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:09.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:09.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:43:10.724 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:43:10.724 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:43:10.724 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:43:10.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:43:10.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:43:10.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:43:11.369 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3259910443,v1:127.0.0.1:6819/3259910443] [v2:127.0.0.1:6820/3259910443,v1:127.0.0.1:6821/3259910443] exists,up ae624dbf-adf1-494c-84f1-2adf091ea1b5 2024-09-01T01:43:11.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:43:11.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:43:11.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:43:11.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-01T01:43:16.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-01T01:43:16.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-01T01:43:16.877 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'foo' already exists 2024-09-01T01:43:16.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T01:43:17.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-01T01:43:17.972 INFO:tasks.workunit.client.0.smithi037.stdout:hints = 1 2024-09-01T01:43:17.973 INFO:tasks.workunit.client.0.smithi037.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-01T01:43:17.973 INFO:tasks.workunit.client.0.smithi037.stdout:Object prefix: benchmark_data_smithi037_35555 2024-09-01T01:43:17.973 INFO:tasks.workunit.client.0.smithi037.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T01:43:17.973 INFO:tasks.workunit.client.0.smithi037.stdout: 0 0 0 0 0 0 - 0 2024-09-01T01:43:18.973 INFO:tasks.workunit.client.0.smithi037.stdout: 1 16 76 60 0.234348 0.234375 0.266449 0.244802 2024-09-01T01:43:19.973 INFO:tasks.workunit.client.0.smithi037.stdout: 2 16 129 113 0.220673 0.207031 0.269864 0.257273 2024-09-01T01:43:20.973 INFO:tasks.workunit.client.0.smithi037.stdout: 3 16 193 177 0.230436 0.25 0.219269 0.256888 2024-09-01T01:43:21.973 INFO:tasks.workunit.client.0.smithi037.stdout: 4 16 257 241 0.235316 0.25 0.25043 0.25573 2024-09-01T01:43:22.973 INFO:tasks.workunit.client.0.smithi037.stdout: 5 16 321 305 0.238248 0.25 0.234973 0.254801 2024-09-01T01:43:23.973 INFO:tasks.workunit.client.0.smithi037.stdout: 6 16 385 369 0.240202 0.25 0.251523 0.252634 2024-09-01T01:43:24.973 INFO:tasks.workunit.client.0.smithi037.stdout: 7 16 459 443 0.247176 0.289062 0.243265 0.251244 2024-09-01T01:43:25.973 INFO:tasks.workunit.client.0.smithi037.stdout: 8 16 523 507 0.247524 0.25 0.229539 0.250219 2024-09-01T01:43:26.974 INFO:tasks.workunit.client.0.smithi037.stdout: 9 16 588 572 0.248229 0.253906 0.212475 0.247916 2024-09-01T01:43:27.974 INFO:tasks.workunit.client.0.smithi037.stdout: 10 16 668 652 0.254651 0.3125 0.19324 0.243207 2024-09-01T01:43:28.974 INFO:tasks.workunit.client.0.smithi037.stdout:Total time run: 10.0952 2024-09-01T01:43:28.974 INFO:tasks.workunit.client.0.smithi037.stdout:Total writes made: 668 2024-09-01T01:43:28.974 INFO:tasks.workunit.client.0.smithi037.stdout:Write size: 4096 2024-09-01T01:43:28.974 INFO:tasks.workunit.client.0.smithi037.stdout:Object size: 4096 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Bandwidth (MB/sec): 0.258477 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev Bandwidth: 0.0285153 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Max bandwidth (MB/sec): 0.3125 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Min bandwidth (MB/sec): 0.207031 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Average IOPS: 66 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev IOPS: 7.29992 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Max IOPS: 80 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Min IOPS: 53 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Average Latency(s): 0.241348 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev Latency(s): 0.0337031 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Max latency(s): 0.497558 2024-09-01T01:43:28.975 INFO:tasks.workunit.client.0.smithi037.stdout:Min latency(s): 0.158535 2024-09-01T01:43:29.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-01T01:43:39.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-01T01:43:39.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:43:39.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr:2' 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:43:39.494 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:43:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-01T01:43:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-01T01:43:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-01T01:43:39.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:43:39.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:43:39.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-01T01:43:39.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-01T01:43:39.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-09-01T01:43:39.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:43:39.725 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:43:39.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-01T01:43:39.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-01T01:43:39.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509452' 2024-09-01T01:43:39.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:43:39.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-01T01:43:39.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:43:39.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:43:39.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-01T01:43:39.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:43:39.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-01T01:43:39.843 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836509 2024-09-01T01:43:39.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-01T01:43:39.844 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:43:40.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-01T01:43:40.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:43:41.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:43:41.280 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:43:41.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836509 2024-09-01T01:43:41.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:43:41.713 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-01T01:43:41.713 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:43:41.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:43:41.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-01T01:43:41.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:43:41.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-01T01:43:41.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-01T01:43:41.716 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672981 2024-09-01T01:43:41.716 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:43:42.149 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-01T01:43:42.149 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:43:42.149 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-01T01:43:42.150 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:43:42.150 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:43:42.151 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-01T01:43:42.151 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:43:42.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-01T01:43:42.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-01T01:43:42.152 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509452 2024-09-01T01:43:42.152 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:43:42.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-01T01:43:42.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:43:42.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:43:42.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 2024-09-01T01:43:42.673 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:43:42.673 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:43:42.673 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:43:42.673 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:43:42.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:43:42.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:43:42.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-01T01:43:42.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:43:42.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:43:42.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:43:42.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:43:42.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:43:42.687 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:43:42.688 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:43:42.688 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:43:42.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:43:42.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:43:42.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 2024-09-01T01:43:42.759 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:43:42.759 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:43:42.759 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:43:42.760 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:43:42.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:43:42.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:43:42.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-01T01:43:42.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:43:42.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:43:42.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:43:42.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:43:42.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 0 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:38 2024", 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.853 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.355, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.361 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.380, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.520, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.380, 2024-09-01T01:43:42.854 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.520, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.222 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:38 2024", 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.294, 2024-09-01T01:43:42.855 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.334, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.334 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.344, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.489, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.344, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.489, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.856 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.191 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:42 2024", 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.276, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.313, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.299 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.857 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.330, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.466, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.330, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.466, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.205 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:42 2024", 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.858 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.252, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.302, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.289 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.303, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.448, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.303, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.448, 2024-09-01T01:43:42.859 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:43:42.860 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.860 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.175 2024-09-01T01:43:42.860 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:43:42.860 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:43:42.860 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:43:42.861 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.861 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:43:42.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-01T01:43:42.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-01T01:43:42.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:43:42.867 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:43:42.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:43:42.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:43:42.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:43:42.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 0 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.940 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.341, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.354, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.373 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.941 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.242, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.242 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.391, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.856, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.856 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.299 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:36 2024", 2024-09-01T01:43:42.942 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.373, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.340, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.307 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.261, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.215, 2024-09-01T01:43:42.943 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.188 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.458, 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.465, 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.465 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.251 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.944 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.339, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.356, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.368 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.248, 2024-09-01T01:43:42.945 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.248, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.248 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.479, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.504, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.504 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.291 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:38 2024", 2024-09-01T01:43:42.946 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.355, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.361 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.259, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.245, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.245 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.947 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.380, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.520, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.264 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.948 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.329, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.345, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.342 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.243, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.243, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.243 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.419, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.445, 2024-09-01T01:43:42.949 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.445 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.237 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:36 2024", 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.334, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.319, 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.298 2024-09-01T01:43:42.950 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.227, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.205, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.181 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.426, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.426, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.508 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.268 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.951 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:38 2024", 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.294, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.334, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.334 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.256, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.231, 2024-09-01T01:43:42.952 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.231 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.344, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.489, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.248 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:43:42.953 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.309, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.312, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.333 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.220, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.213, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.213 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.954 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.437, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.506, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.506 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.229 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.308, 2024-09-01T01:43:42.955 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.311, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.325 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.219, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.219, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.219 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.461, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.467, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.467 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.271 2024-09-01T01:43:42.956 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.281, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.325, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.303 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.211, 2024-09-01T01:43:42.957 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.211, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.200 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.367, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.493, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.284 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:43:42.958 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.324, 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.307, 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.319 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.218, 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.195, 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.195 2024-09-01T01:43:42.959 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.442, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.804, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.804 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.250 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:43:37 2024", 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:43:42.960 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.278, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.320, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.292 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.221, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.203, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.188 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.386, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.470, 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:43:42.961 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:43:42.962 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.303 2024-09-01T01:43:42.962 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:43:42.962 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:43:42.962 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:43:42.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-01T01:44:02.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-01T01:44:02.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:44:02.956 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr:2' 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:03.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:44:03.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-01T01:44:03.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-01T01:44:03.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-01T01:44:03.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:03.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:44:03.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-01T01:44:03.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-01T01:44:03.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672988' 2024-09-01T01:44:03.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:03.599 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:44:03.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-01T01:44:03.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-01T01:44:03.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672988 2-64424509459' 2024-09-01T01:44:03.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:03.715 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-01T01:44:03.715 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:03.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:44:03.716 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-01T01:44:03.716 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:03.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-01T01:44:03.717 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836516 2024-09-01T01:44:03.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-01T01:44:03.717 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:44:04.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-01T01:44:04.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:04.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-01T01:44:04.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:44:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-01T01:44:04.145 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:04.145 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-01T01:44:04.146 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672988 2024-09-01T01:44:04.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-01T01:44:04.146 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:44:04.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672986 -lt 42949672988 2024-09-01T01:44:04.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:44:05.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:44:05.587 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:44:06.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-01T01:44:06.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:06.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-01T01:44:06.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:06.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:44:06.039 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-01T01:44:06.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:06.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-01T01:44:06.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-01T01:44:06.041 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509459 2024-09-01T01:44:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:44:06.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-01T01:44:06.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:44:06.472 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:44:06.472 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:44:06.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 2024-09-01T01:44:06.552 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:44:06.552 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:44:06.552 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:44:06.553 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:44:06.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:44:06.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:44:06.564 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-01T01:44:06.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:44:06.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:44:06.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:44:06.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:44:06.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:44:06.567 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:44:06.567 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:44:06.567 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:44:06.568 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:44:06.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:44:06.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 2024-09-01T01:44:06.639 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:44:06.640 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:44:06.640 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:44:06.640 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:44:06.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:44:06.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:44:06.650 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.650 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-01T01:44:06.652 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:44:06.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 0 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:03 2024", 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.734 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.324, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.322, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.347 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.405, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.410, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.405, 2024-09-01T01:44:06.735 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.410, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.283 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:03 2024", 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:44:06.736 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.302, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.323 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.373, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.446, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.373, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.446, 2024-09-01T01:44:06.737 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.239 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.308, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.292, 2024-09-01T01:44:06.738 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.292 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.343, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.436, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.343, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.436, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.221 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.739 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.297, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.298 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.740 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.382, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.464, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.382, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.464, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.255 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:44:06.741 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:44:06.742 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.742 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:44:06.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-01T01:44:06.744 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.744 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-01T01:44:06.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:44:06.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:44:06.748 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:44:06.748 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:44:06.748 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:44:06.748 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:44:06.748 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:44:06.749 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:44:06.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:44:06.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:44:06.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 0 2024-09-01T01:44:06.820 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:44:06.820 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:44:06.820 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.821 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.393, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.346, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.359 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.312, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.238, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.238 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.484, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.510, 2024-09-01T01:44:06.822 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.510 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.299 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.823 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.367, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.330, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.339 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.255, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.209, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.209 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.824 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.440, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.463, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.463 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.247 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.825 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.320, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.314, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.350 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.238, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.217, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.217 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.403, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.470, 2024-09-01T01:44:06.826 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.856 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.293 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:03 2024", 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.827 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.324, 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.322, 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.347 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.828 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.285, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.222, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.222 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.405, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.410, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.520 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.295 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.829 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:01 2024", 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.326, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.309, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.309 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.257, 2024-09-01T01:44:06.830 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.207, 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.188 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.439, 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.471, 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.471 2024-09-01T01:44:06.831 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.832 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.262 2024-09-01T01:44:06.832 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.832 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.832 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:03 2024", 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.302, 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.323 2024-09-01T01:44:06.833 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.234, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.191, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.191 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.373, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.446, 2024-09-01T01:44:06.834 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.489 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.247 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:44:06.835 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.299, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.286, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.314 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.253, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.213, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.213 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.836 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.371, 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.450, 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.506 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.243 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:01 2024", 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.837 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.313, 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.304, 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.299 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.838 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.225, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.198, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.181 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.425, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.444, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.508 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.233 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.839 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.278, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.287, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.310 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.215, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.215, 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.215 2024-09-01T01:44:06.840 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.381, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.488, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.488 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.226 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.841 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.308, 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.292, 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.292 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.842 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.255, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.175, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.175 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.343, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.436, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.470 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.261 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.843 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.302, 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.297, 2024-09-01T01:44:06.844 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.298 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.250, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.205, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.200 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.382, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.464, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.227 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.845 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:02 2024", 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.284, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.271, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.301 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.215, 2024-09-01T01:44:06.846 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.192, 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.192 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.347, 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.437, 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.804 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.261 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:44:06.847 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:44:06.848 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:44:06.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-01T01:44:56.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-01T01:44:56.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:44:56.836 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:44:57.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:44:57.282 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:44:57.282 INFO:tasks.workunit.client.0.smithi037.stderr:2' 2024-09-01T01:44:57.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:44:57.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:57.282 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:44:57.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-01T01:44:57.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-01T01:44:57.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-01T01:44:57.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:57.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:44:57.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673001 2024-09-01T01:44:57.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673001 2024-09-01T01:44:57.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-42949673001' 2024-09-01T01:44:57.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:44:57.513 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:44:57.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509472 2024-09-01T01:44:57.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509472 2024-09-01T01:44:57.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-42949673001 2-64424509472' 2024-09-01T01:44:57.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:57.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-01T01:44:57.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:57.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:44:57.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-01T01:44:57.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:57.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-01T01:44:57.632 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836529 2024-09-01T01:44:57.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-01T01:44:57.633 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:44:58.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-01T01:44:58.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:58.059 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673001 2024-09-01T01:44:58.059 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:58.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:44:58.060 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673001 2024-09-01T01:44:58.060 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:58.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673001 2024-09-01T01:44:58.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-01T01:44:58.062 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949673001 2024-09-01T01:44:58.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:44:58.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673001 -lt 42949673001 2024-09-01T01:44:58.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:44:58.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509472 2024-09-01T01:44:58.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:44:58.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:44:58.476 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509472 2024-09-01T01:44:58.476 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:44:58.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509472 2024-09-01T01:44:58.477 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509472 2024-09-01T01:44:58.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-01T01:44:58.478 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:44:58.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509472 2024-09-01T01:44:58.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:44:59.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:44:59.911 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:45:00.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:45:00.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 2024-09-01T01:45:00.441 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.442 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:45:00.442 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:45:00.442 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:45:00.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:45:00.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.455 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:45:00.455 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.456 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:45:00.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 2024-09-01T01:45:00.529 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.529 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 1000, 2024-09-01T01:45:00.529 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:45:00.529 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.537 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.537 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:45:00.539 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:45:00.540 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.540 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-01T01:45:00.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.543 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 0 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:58 2024", 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.624 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.319, 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.352, 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.336 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.404, 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.477, 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.522 2024-09-01T01:45:00.625 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.628 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.628 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.404, 2024-09-01T01:45:00.628 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.477, 2024-09-01T01:45:00.628 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.522 2024-09-01T01:45:00.628 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.301 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:58 2024", 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.331, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.351, 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.325 2024-09-01T01:45:00.629 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.386, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.493, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.386, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.493, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.285 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.630 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.346, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.339, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.319 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.631 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.419, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.507, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.507 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.419, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.507, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.507 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.269 2024-09-01T01:45:00.632 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.336, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.324, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.308 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.633 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.372, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.428, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.468 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.372, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.428, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.468 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.251 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:45:00.634 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:45:00.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-01T01:45:00.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:45:00.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 0 2024-09-01T01:45:00.710 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 0, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [ 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:52 2024", 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.382, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.327, 2024-09-01T01:45:00.711 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.323 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.237, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.211, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.209 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.470, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.499, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.499 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.258 2024-09-01T01:45:00.712 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:52 2024", 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.378, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.351, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.347 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.307, 2024-09-01T01:45:00.713 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.230, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.230 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.455, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.512, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.512 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.325 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:53 2024", 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.714 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.331, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.348, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.336 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.244, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.244, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.222 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.715 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.417, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.477, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.522 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.250 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.716 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.346, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.339, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.319 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.219, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.219, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.205 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.419, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.507, 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.507 2024-09-01T01:45:00.717 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.219 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:53 2024", 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 2, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.344, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.341, 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.322 2024-09-01T01:45:00.718 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.265, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.232, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.191 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.442, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.493, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.493 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.295 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.719 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:56 2024", 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.303, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.340, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.326 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.243, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.243, 2024-09-01T01:45:00.720 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.210 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.397, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.477, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.501 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.277 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 0, 2024-09-01T01:45:00.721 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.336, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.324, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.308 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.272, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.225, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.202 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.372, 2024-09-01T01:45:00.722 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.428, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.468 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.272 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.318, 2024-09-01T01:45:00.723 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.332, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.328 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.237, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.225, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.225 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.430, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.476, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.533 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.266 2024-09-01T01:45:00.724 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 1, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.294, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.320, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.310 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.220, 2024-09-01T01:45:00.725 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.220, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.210 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.412, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.441, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.514 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.228 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:45:00.726 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "front", 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.317, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.320, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.308 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.240, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.200, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.200 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.448, 2024-09-01T01:45:00.727 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.448, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.488 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.244 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:56 2024", 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 1, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.304, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.308, 2024-09-01T01:45:00.728 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.302 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.235, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.209, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.190 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.360, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.421, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.426 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.295 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.729 INFO:tasks.workunit.client.0.smithi037.stdout: { 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "last update": "Sun Sep 1 01:44:57 2024", 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "stale": false, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "from osd": 2, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "to osd": 0, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "interface": "back", 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "average": { 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.277, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.299, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.293 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "min": { 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.205, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.205, 2024-09-01T01:45:00.730 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.196 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: "max": { 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: "1min": 0.397, 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: "5min": 0.461, 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: "15min": 0.461 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: }, 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: "last": 0.210 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: } 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout: ] 2024-09-01T01:45:00.731 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-01T01:45:00.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-01T01:45:00.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mgr.x.asok 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-mgr.x.asok dump_osd_network 99 2024-09-01T01:45:00.797 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.797 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 99, 2024-09-01T01:45:00.798 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:45:00.798 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.805 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.805 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-01T01:45:00.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-01T01:45:00.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-osd.0.asok 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-01T01:45:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.30030/ceph-osd.0.asok dump_osd_network 98 2024-09-01T01:45:00.888 INFO:tasks.workunit.client.0.smithi037.stdout:{ 2024-09-01T01:45:00.888 INFO:tasks.workunit.client.0.smithi037.stdout: "threshold": 98, 2024-09-01T01:45:00.888 INFO:tasks.workunit.client.0.smithi037.stdout: "entries": [] 2024-09-01T01:45:00.888 INFO:tasks.workunit.client.0.smithi037.stdout:} 2024-09-01T01:45:00.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-01T01:45:00.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-01T01:45:00.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-01T01:45:00.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-01T01:45:00.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-01T01:45:00.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-01T01:45:00.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:45:00.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-01T01:45:00.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:45:00.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:45:00.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:45:00.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:45:00.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:45:01.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:45:01.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:45:01.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:45:01.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:45:01.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:45:01.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:45:01.010 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:45:01.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:45:01.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:45:01.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:45:01.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:45:01.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:45:01.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:45:01.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-01T01:45:01.032 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:45:01.032 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.032 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30030 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:45:01.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:45:01.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:45:01.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:45:01.036 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:45:01.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:45:01.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:45:01.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:45:01.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:45:01.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:45:01.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:45:01.039 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:45:01.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:45:01.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:45:01.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-01T01:45:01.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:45:01.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30030 2024-09-01T01:45:01.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:45:01.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:45:01.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2024-09-01T01:45:01.044 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:45:01.044 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.044 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.044 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.30030 2024-09-01T01:45:01.046 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:45:01.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:45:01.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:45:01.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-01T01:45:01.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2024-09-01T01:45:01.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:01.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:45:01.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:45:01.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:45:01.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:45:01.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:45:01.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:45:01.710 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:45:01.710 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mon.a.asok 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:45:01.711 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30030/ceph-mon.a.asok config get fsid 2024-09-01T01:45:01.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.30030/ceph-mon.a.asok 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:45:01.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.30030/ceph-mon.a.asok config get mon_host 2024-09-01T01:45:01.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-01T01:45:01.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2024-09-01T01:45:01.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:45:01.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:45:01.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:45:01.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2024-09-01T01:45:01.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:02.733 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:02.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:02.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:45:02.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:45:02.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-01T01:45:02.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:45:02.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:45:02.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:45:02.767 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:45:02.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=df2bd34e-d42c-4182-9c52-30799f3de9d1 2024-09-01T01:45:02.769 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 df2bd34e-d42c-4182-9c52-30799f3de9d1 2024-09-01T01:45:02.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 df2bd34e-d42c-4182-9c52-30799f3de9d1' 2024-09-01T01:45:02.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:45:02.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAex9NmZNHYLhAA49l/+H9sm6b8njlWBQlywQ== 2024-09-01T01:45:02.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAex9NmZNHYLhAA49l/+H9sm6b8njlWBQlywQ=="}' 2024-09-01T01:45:02.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new df2bd34e-d42c-4182-9c52-30799f3de9d1 -i td/network-ping/0/new.json 2024-09-01T01:45:03.058 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:45:03.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2024-09-01T01:45:03.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAex9NmZNHYLhAA49l/+H9sm6b8njlWBQlywQ== --osd-uuid df2bd34e-d42c-4182-9c52-30799f3de9d1 2024-09-01T01:45:03.101 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:03.100+0000 7f9bc8a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:03.126 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:03.125+0000 7f9bc8a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:03.127 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:03.127+0000 7f9bc8a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:03.127 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:03.127+0000 7f9bc8a2c640 -1 bdev(0x558b415cfc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:45:03.127 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:03.127+0000 7f9bc8a2c640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-01T01:45:09.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-01T01:45:09.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:45:09.041 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:45:09.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:45:09.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:45:09.603 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:45:09.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:45:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:45:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:45:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:45:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:45:09.632 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:09.631+0000 7fcb8020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:09.637 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:09.634+0000 7fcb8020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:09.637 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:09.635+0000 7fcb8020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:45:10.052 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:45:10.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:10.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:10.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:11.005 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:11.005+0000 7fcb8020b640 -1 Falling back to public interface 2024-09-01T01:45:11.477 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:11.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:11.478 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:45:11.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:45:11.479 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:11.479 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:11.919 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:12.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:12.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:12.921 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:45:12.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:45:12.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:12.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:13.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:13.840 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:13.840+0000 7fcb8020b640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:45:14.332 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:14.332 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:14.332 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:45:14.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:45:14.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:14.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:14.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:15.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:15.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:15.886 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:45:15.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:45:15.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:15.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:16.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:17.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:17.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:17.298 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:45:17.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:45:17.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:17.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:17.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:18.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:18.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:18.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:45:18.740 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:45:18.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:18.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:19.151 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:20.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:20.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:20.153 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:45:20.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:45:20.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:20.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:20.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:21.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:21.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:21.719 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:45:21.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:45:21.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:21.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:22.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:23.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:23.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:23.192 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:45:23.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:45:23.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:23.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:23.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:24.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:24.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:24.705 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:45:24.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:45:24.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:24.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:25.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:26.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:26.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:26.125 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:45:26.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:45:26.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:26.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:26.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:27.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:27.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:27.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:45:27.666 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:45:27.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:27.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:28.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:29.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:29.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:29.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:45:29.353 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:45:29.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:29.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:30.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:31.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:31.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:31.320 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:45:31.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:45:31.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:31.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:32.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:33.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:33.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:33.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:45:33.049 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:45:33.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:33.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:33.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:34.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:34.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:34.906 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:45:34.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:45:34.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:34.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:35.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:36.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:36.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:36.952 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:45:36.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:45:36.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:36.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:45:37.390 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/943874029,v1:127.0.0.1:6803/943874029] [v2:127.0.0.1:6804/943874029,v1:127.0.0.1:6805/943874029] exists,up df2bd34e-d42c-4182-9c52-30799f3de9d1 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2024-09-01T01:45:37.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:45:37.396 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:45:37.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1de44483-95aa-473d-9613-4c5ad77a0b58 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1de44483-95aa-473d-9613-4c5ad77a0b58' 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 1de44483-95aa-473d-9613-4c5ad77a0b58 2024-09-01T01:45:37.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:45:37.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBx9NmJSufGBAAofRkIQx6OvQgjRZCOjv7JA== 2024-09-01T01:45:37.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBx9NmJSufGBAAofRkIQx6OvQgjRZCOjv7JA=="}' 2024-09-01T01:45:37.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1de44483-95aa-473d-9613-4c5ad77a0b58 -i td/network-ping/1/new.json 2024-09-01T01:45:37.939 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:45:37.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2024-09-01T01:45:37.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBx9NmJSufGBAAofRkIQx6OvQgjRZCOjv7JA== --osd-uuid 1de44483-95aa-473d-9613-4c5ad77a0b58 2024-09-01T01:45:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:37.981+0000 7f0d05c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:37.983 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:37.983+0000 7f0d05c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:37.984 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:37.984+0000 7f0d05c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:37.985 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:37.984+0000 7f0d05c87640 -1 bdev(0x564aa3199c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-01T01:45:37.985 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:37.984+0000 7f0d05c87640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-01T01:45:43.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-01T01:45:43.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:45:43.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T01:45:43.925 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T01:45:43.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:45:44.484 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T01:45:44.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T01:45:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:45:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:45:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:45:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:45:44.513 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:44.512+0000 7fd5a080f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:44.518 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:44.517+0000 7fd5a080f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:44.519 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:44.519+0000 7fd5a080f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:45:44.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:44.966 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:45:44.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:45:44.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:44.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:45.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:45.632 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:45.632+0000 7fd5a080f640 -1 Falling back to public interface 2024-09-01T01:45:46.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:46.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:46.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:45:46.418 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:45:46.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:46.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:46.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:47.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:47.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:47.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:45:47.854 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:45:47.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:47.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:48.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:48.422 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:45:48.421+0000 7fd5a080f640 -1 osd.1 0 log_to_monitors true 2024-09-01T01:45:49.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:49.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:49.370 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:45:49.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:45:49.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:49.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:49.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:50.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:50.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:50.834 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:45:50.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:45:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:51.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:45:52.339 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:45:52.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:52.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:52.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:53.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:53.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:53.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:45:53.774 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:45:53.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:53.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:54.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:55.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:55.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:55.210 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:45:55.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:45:55.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:55.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:55.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:56.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:56.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:56.745 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:45:56.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:45:56.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:56.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:57.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:58.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:58.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:58.156 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:45:58.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:45:58.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:58.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:45:58.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:45:59.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:45:59.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:45:59.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:45:59.578 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:45:59.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:45:59.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:00.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:01.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:01.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:01.021 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:46:01.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:46:01.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:01.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:01.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:02.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:02.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:02.441 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:46:02.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:46:02.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:02.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:02.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:46:03.894 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:46:03.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:03.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:04.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:05.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:05.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:05.520 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:46:05.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:46:05.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:05.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:06.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:07.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:07.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:07.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:46:07.445 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:46:07.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:07.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:08.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:09.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:09.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:09.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:46:09.141 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:46:09.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:09.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:09.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:10.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:10.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:10.794 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:46:10.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:46:10.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:10.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:46:11.210 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1866732233,v1:127.0.0.1:6811/1866732233] [v2:127.0.0.1:6812/1866732233,v1:127.0.0.1:6813/1866732233] exists,up 1de44483-95aa-473d-9613-4c5ad77a0b58 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2024-09-01T01:46:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:46:11.215 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:46:11.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=19be6336-36a1-467e-a9aa-2faf9b55bad0 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 19be6336-36a1-467e-a9aa-2faf9b55bad0' 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 19be6336-36a1-467e-a9aa-2faf9b55bad0 2024-09-01T01:46:11.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:46:11.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBjx9NmiE3XDRAADn6OWovkX8Q8o0ENFXncVA== 2024-09-01T01:46:11.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBjx9NmiE3XDRAADn6OWovkX8Q8o0ENFXncVA=="}' 2024-09-01T01:46:11.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 19be6336-36a1-467e-a9aa-2faf9b55bad0 -i td/network-ping/2/new.json 2024-09-01T01:46:11.857 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:46:11.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2024-09-01T01:46:11.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjx9NmiE3XDRAADn6OWovkX8Q8o0ENFXncVA== --osd-uuid 19be6336-36a1-467e-a9aa-2faf9b55bad0 2024-09-01T01:46:11.900 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:11.899+0000 7f8b570b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:11.902 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:11.901+0000 7f8b570b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:11.903 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:11.902+0000 7f8b570b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:11.904 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:11.902+0000 7f8b570b6640 -1 bdev(0x55c0de23dc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-01T01:46:11.904 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:11.903+0000 7f8b570b6640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-01T01:46:18.005 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-01T01:46:18.005 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:46:18.006 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-01T01:46:18.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T01:46:18.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:46:18.571 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-01T01:46:18.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T01:46:18.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=c9da8893-2e7e-4650-9ae0-58c7a11d7145 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.30030/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:46:18.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:46:18.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:46:18.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:46:18.600 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:18.599+0000 7f6187610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:18.603 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:18.602+0000 7f6187610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:18.603 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:18.603+0000 7f6187610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:46:19.021 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:46:19.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:19.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:19.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:19.948 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:19.948+0000 7f6187610640 -1 Falling back to public interface 2024-09-01T01:46:20.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:20.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:20.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:46:20.501 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:46:20.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:20.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:20.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:21.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:21.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:21.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:46:21.968 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:46:21.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:21.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:22.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:23.232 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:46:23.232+0000 7f6187610640 -1 osd.2 0 log_to_monitors true 2024-09-01T01:46:23.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:23.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:23.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:46:23.397 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:46:23.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:23.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:23.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:24.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:24.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:24.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:46:24.962 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:46:24.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:24.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:25.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:26.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:26.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:26.500 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:46:26.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:46:26.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:26.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:26.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:27.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:27.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:27.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:46:27.914 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:46:27.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:27.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:28.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:29.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:29.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:29.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:46:29.372 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:46:29.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:29.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:29.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:30.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:30.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:30.802 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:46:30.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:46:30.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:30.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:31.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:32.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:32.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:32.281 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:46:32.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:46:32.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:32.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:32.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:33.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:33.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:33.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:46:33.694 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:46:33.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:33.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:34.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:35.167 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:35.167 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:35.167 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:46:35.167 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:46:35.168 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:35.168 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:35.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:36.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:36.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:36.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:46:36.691 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:46:36.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:36.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:37.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:38.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:38.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:38.283 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:46:38.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:46:38.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:38.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:38.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:39.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:39.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:39.911 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:46:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:46:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:40.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:41.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:41.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:41.598 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:46:41.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:46:41.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:41.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:42.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:43.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:43.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:43.596 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:46:43.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:46:43.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:43.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:44.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:45.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:45.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:45.405 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:46:45.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:46:45.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:45.406 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:45.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:46:46.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:46:46.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:46:46.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T01:46:46.879 INFO:tasks.workunit.client.0.smithi037.stdout:18 2024-09-01T01:46:46.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:46:46.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:46:47.298 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1804889725,v1:127.0.0.1:6819/1804889725] [v2:127.0.0.1:6820/1804889725,v1:127.0.0.1:6821/1804889725] exists,up 19be6336-36a1-467e-a9aa-2faf9b55bad0 2024-09-01T01:46:47.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:46:47.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:46:47.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:46:47.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-01T01:46:52.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-01T01:46:52.814 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'dc1' already exists 2024-09-01T01:46:52.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-01T01:46:53.290 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'dc2' already exists 2024-09-01T01:46:53.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-01T01:46:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'dc3' already exists 2024-09-01T01:46:53.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-01T01:46:54.303 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'rack1' already exists 2024-09-01T01:46:54.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-01T01:46:54.817 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'rack2' already exists 2024-09-01T01:46:54.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-01T01:46:55.451 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'rack3' already exists 2024-09-01T01:46:55.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-01T01:46:56.014 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'host1' already exists 2024-09-01T01:46:56.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-01T01:46:56.520 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'host2' already exists 2024-09-01T01:46:56.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-01T01:46:57.074 INFO:tasks.workunit.client.0.smithi037.stderr:bucket 'host3' already exists 2024-09-01T01:46:57.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-01T01:46:57.853 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-01T01:46:57.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-01T01:46:58.413 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-01T01:46:58.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-01T01:46:58.944 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-01T01:46:58.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-01T01:46:59.570 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-01T01:46:59.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-01T01:47:00.163 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-01T01:47:00.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-01T01:47:00.742 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-01T01:47:00.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-01T01:47:01.320 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-01T01:47:01.331 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-01T01:47:01.967 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-01T01:47:01.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-01T01:47:02.474 INFO:tasks.workunit.client.0.smithi037.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-01T01:47:02.485 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-01T01:47:02.979 INFO:tasks.workunit.client.0.smithi037.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-01T01:47:02.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-01T01:47:03.532 INFO:tasks.workunit.client.0.smithi037.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-01T01:47:03.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-01T01:47:04.110 INFO:tasks.workunit.client.0.smithi037.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-01T01:47:04.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-01T01:47:04.661 INFO:tasks.workunit.client.0.smithi037.stderr:rule myrule already exists 2024-09-01T01:47:04.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-01T01:47:04.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-01T01:47:05.179 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'foo' already exists 2024-09-01T01:47:05.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T01:47:06.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-01T01:47:06.262 INFO:tasks.workunit.client.0.smithi037.stdout:hints = 1 2024-09-01T01:47:06.262 INFO:tasks.workunit.client.0.smithi037.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-01T01:47:06.262 INFO:tasks.workunit.client.0.smithi037.stdout:Object prefix: benchmark_data_smithi037_42412 2024-09-01T01:47:06.262 INFO:tasks.workunit.client.0.smithi037.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T01:47:06.262 INFO:tasks.workunit.client.0.smithi037.stdout: 0 0 0 0 0 0 - 0 2024-09-01T01:47:07.262 INFO:tasks.workunit.client.0.smithi037.stdout: 1 16 59 43 0.16795 0.167969 0.260127 0.323027 2024-09-01T01:47:08.262 INFO:tasks.workunit.client.0.smithi037.stdout: 2 16 117 101 0.197242 0.226562 0.277889 0.292177 2024-09-01T01:47:09.262 INFO:tasks.workunit.client.0.smithi037.stdout: 3 16 178 162 0.210911 0.238281 0.232594 0.285027 2024-09-01T01:47:10.262 INFO:tasks.workunit.client.0.smithi037.stdout: 4 16 242 226 0.220674 0.25 0.225678 0.273455 2024-09-01T01:47:11.263 INFO:tasks.workunit.client.0.smithi037.stdout: 5 16 313 297 0.231999 0.277344 0.248925 0.264961 2024-09-01T01:47:12.263 INFO:tasks.workunit.client.0.smithi037.stdout: 6 16 377 361 0.234993 0.25 0.223789 0.262327 2024-09-01T01:47:13.263 INFO:tasks.workunit.client.0.smithi037.stdout: 7 16 434 418 0.233227 0.222656 0.277937 0.262209 2024-09-01T01:47:14.263 INFO:tasks.workunit.client.0.smithi037.stdout: 8 16 505 489 0.238738 0.277344 0.222621 0.259432 2024-09-01T01:47:15.263 INFO:tasks.workunit.client.0.smithi037.stdout: 9 16 589 573 0.248665 0.328125 0.146027 0.24842 2024-09-01T01:47:16.263 INFO:tasks.workunit.client.0.smithi037.stdout: 10 16 665 649 0.253482 0.296875 0.168748 0.243141 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Total time run: 10.1089 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Total writes made: 665 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Write size: 4096 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Object size: 4096 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Bandwidth (MB/sec): 0.256968 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev Bandwidth: 0.0445172 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Max bandwidth (MB/sec): 0.328125 2024-09-01T01:47:17.264 INFO:tasks.workunit.client.0.smithi037.stdout:Min bandwidth (MB/sec): 0.167969 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Average IOPS: 65 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev IOPS: 11.3964 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Max IOPS: 84 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Min IOPS: 43 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Average Latency(s): 0.242827 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Stddev Latency(s): 0.0511462 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Max latency(s): 0.584152 2024-09-01T01:47:17.265 INFO:tasks.workunit.client.0.smithi037.stdout:Min latency(s): 0.130347 2024-09-01T01:47:17.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-01T01:47:27.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-01T01:47:27.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:47:27.343 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:47:27.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:47:27.782 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:47:27.783 INFO:tasks.workunit.client.0.smithi037.stderr:2' 2024-09-01T01:47:27.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:47:27.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:47:27.783 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:47:27.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-01T01:47:27.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-01T01:47:27.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-01T01:47:27.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:47:27.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:47:28.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-01T01:47:28.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-01T01:47:28.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672981' 2024-09-01T01:47:28.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:47:28.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672981 2-64424509456' 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:47:28.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:47:28.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-01T01:47:28.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:47:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-01T01:47:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-01T01:47:28.214 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836506 2024-09-01T01:47:28.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:47:28.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-01T01:47:28.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:47:29.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:47:29.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:47:30.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-01T01:47:30.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:47:30.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-01T01:47:30.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:47:30.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:47:30.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-01T01:47:30.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:47:30.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-01T01:47:30.100 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672981 2024-09-01T01:47:30.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-01T01:47:30.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:47:30.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-01T01:47:30.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:47:30.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-01T01:47:30.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:47:30.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:47:30.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-01T01:47:30.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:47:30.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-01T01:47:30.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-01T01:47:30.535 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509456 2024-09-01T01:47:30.535 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:47:30.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-01T01:47:30.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-01T01:47:30.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-01T01:47:31.412 INFO:tasks.workunit.client.0.smithi037.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.346ms); Slow OSD heartbeats on front (longest 0.345ms); 1 pool(s) do not have an application enabled 2024-09-01T01:47:31.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-01T01:47:31.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-01T01:47:31.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-01T01:47:31.958 INFO:tasks.workunit.client.0.smithi037.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.346ms); Slow OSD heartbeats on front (longest 0.345ms); 1 pool(s) do not have an application enabled 2024-09-01T01:47:31.958 INFO:tasks.workunit.client.0.smithi037.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.346ms) 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.346 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.328 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.323 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.322 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.299 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.297 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.345ms) 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.345 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.344 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.341 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.339 msec 2024-09-01T01:47:31.959 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.327 msec 2024-09-01T01:47:31.960 INFO:tasks.workunit.client.0.smithi037.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.327 msec 2024-09-01T01:47:31.960 INFO:tasks.workunit.client.0.smithi037.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-01T01:47:31.960 INFO:tasks.workunit.client.0.smithi037.stdout: application not enabled on pool 'foo' 2024-09-01T01:47:31.960 INFO:tasks.workunit.client.0.smithi037.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-01T01:47:31.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-01T01:47:31.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-01T01:47:31.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-01T01:47:31.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:47:31.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:47:31.975 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:47:32.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:47:32.081 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:47:32.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:47:32.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:47:32.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:47:32.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:47:32.084 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:47:32.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:47:32.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:47:32.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:47:32.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:47:32.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-01T01:47:32.106 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:47:32.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:47:32.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30030 2024-09-01T01:47:32.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:47:32.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/network-ping 0 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:47:32.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:47:32.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:47:32.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:47:32.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:47:32.111 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:47:32.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:47:32.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:47:32.112 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:47:32.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:47:32.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:47:32.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:47:32.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T01:47:32.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2024-09-01T01:47:32.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:47:32.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.30030 2024-09-01T01:47:32.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.30030 2024-09-01T01:47:32.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:47:32.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:47:32.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T01:47:32.118 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T01:47:32.118 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T01:47:32.143 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-01T01:47:32.143 DEBUG:teuthology.orchestra.run.smithi037:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-01T01:47:32.206 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T01:47:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T01:47:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ok-to-stop 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-01T01:47:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none ' 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none ' 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none ' 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-01T01:47:32.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T01:47:32.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-01T01:47:32.217 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_0_osd 2024-09-01T01:47:32.217 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_1_mon_checks 2024-09-01T01:47:32.217 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_2_mons_checks 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_3_mons_checks 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_4_mons_checks 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:TEST_5_mons_checks' 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:47:32.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:47:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:47:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:47:32.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:47:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:47:32.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:47:32.220 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:47:32.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:47:32.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:47:32.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:47:32.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:47:32.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:32.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:47:32.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:47:32.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:47:32.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:47:32.224 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:47:32.224 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.224 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:32.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:47:32.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:47:32.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:32.666 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:47:32.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:47:32.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:47:32.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:47:32.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:47:32.864 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:47:32.865 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:47:32.945 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:47:32.945 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:47:32.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:47:32.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:47:32.946 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:47:32.946 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:47:32.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:47:32.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:47:32.948 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:47:33.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-01T01:47:33.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2024-09-01T01:47:33.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:47:33.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:47:33.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:47:33.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2024-09-01T01:47:33.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:33.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:33.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:47:33.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:47:33.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:47:33.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-01T01:47:33.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:47:33.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:47:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:47:33.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:47:33.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2024-09-01T01:47:33.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:47:33.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f3a31b35-9260-4b63-a301-78d22c4de6ae 2024-09-01T01:47:33.934 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 f3a31b35-9260-4b63-a301-78d22c4de6ae 2024-09-01T01:47:33.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f3a31b35-9260-4b63-a301-78d22c4de6ae' 2024-09-01T01:47:33.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:47:33.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1x9NmEgd3OBAAo0XRcaugk7IIJM8gXbQPbQ== 2024-09-01T01:47:33.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1x9NmEgd3OBAAo0XRcaugk7IIJM8gXbQPbQ=="}' 2024-09-01T01:47:33.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f3a31b35-9260-4b63-a301-78d22c4de6ae -i td/ok-to-stop/0/new.json 2024-09-01T01:47:34.225 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:47:34.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2024-09-01T01:47:34.235 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1x9NmEgd3OBAAo0XRcaugk7IIJM8gXbQPbQ== --osd-uuid f3a31b35-9260-4b63-a301-78d22c4de6ae 2024-09-01T01:47:34.262 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:34.262+0000 7fca5a631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:34.294 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:34.293+0000 7fca5a631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:34.295 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:34.295+0000 7fca5a631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:34.296 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:34.295+0000 7fca5a631640 -1 bdev(0x5565e51fdc00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:47:34.296 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:34.295+0000 7fca5a631640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-01T01:47:40.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-01T01:47:40.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:47:40.081 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:47:40.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:47:40.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:47:40.642 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:47:40.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:47:40.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:47:40.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:47:40.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:47:40.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:47:40.667 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:40.667+0000 7feee2a42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:40.673 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:40.673+0000 7feee2a42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:40.675 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:40.674+0000 7feee2a42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:41.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:47:41.089 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:47:41.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:41.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:41.539 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:42.008 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:42.008+0000 7feee2a42640 -1 Falling back to public interface 2024-09-01T01:47:42.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:42.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:42.540 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:47:42.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:47:42.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:42.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:42.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:43.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:43.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:43.982 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:47:43.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:47:43.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:43.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:44.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:45.162 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:47:45.161+0000 7feee2a42640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:47:45.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:45.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:45.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:47:45.427 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:47:45.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:45.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:45.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:46.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:46.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:46.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:47:46.908 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:47:46.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:46.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:47.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:48.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:48.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:48.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:47:48.320 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:47:48.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:48.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:48.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:49.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:49.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:49.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:47:49.728 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:47:49.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:49.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:50.155 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:51.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:51.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:51.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:47:51.157 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:47:51.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:51.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:51.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:52.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:52.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:52.575 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:47:52.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:47:52.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:52.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:53.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:54.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:54.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:54.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:47:54.068 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:47:54.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:54.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:54.490 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:55.491 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:55.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:55.492 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:47:55.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:47:55.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:55.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:56.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:57.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:57.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:57.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:47:57.038 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:47:57.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:57.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:57.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:47:58.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:47:58.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:47:58.668 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:47:58.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:47:58.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:47:58.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:47:59.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:00.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:00.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:00.429 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:48:00.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:48:00.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:00.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:01.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:02.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:02.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:02.642 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:48:02.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:48:02.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:02.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:03.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:04.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:04.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:04.687 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:48:04.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:48:04.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:04.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:05.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:06.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:06.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:06.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:48:06.310 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:48:06.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:06.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:07.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:08.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:48:08.226 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:48:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:08.648 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/440447324,v1:127.0.0.1:6803/440447324] [v2:127.0.0.1:6804/440447324,v1:127.0.0.1:6805/440447324] exists,up f3a31b35-9260-4b63-a301-78d22c4de6ae 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2024-09-01T01:48:08.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:08.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:48:08.674 INFO:tasks.workunit.client.0.smithi037.stdout:starting mds.a at 2024-09-01T01:48:08.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-01T01:48:09.197 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'meta' already exists 2024-09-01T01:48:09.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-01T01:48:09.795 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'data' already exists 2024-09-01T01:48:09.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-01T01:48:10.685 INFO:tasks.workunit.client.0.smithi037.stderr:filesystem 'myfs' already exists 2024-09-01T01:48:10.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-01T01:48:15.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-01T01:48:16.063 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-01T01:48:16.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-01T01:48:16.418 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-01T01:48:16.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-01T01:48:16.957 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [dne] 2024-09-01T01:48:16.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-01T01:48:16.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2024-09-01T01:48:16.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:16.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:48:17.001 INFO:tasks.workunit.client.0.smithi037.stdout:starting mds.b at 2024-09-01T01:48:17.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-01T01:48:22.005 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-01T01:48:22.554 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [a] 2024-09-01T01:48:22.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-01T01:48:23.119 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [b] 2024-09-01T01:48:23.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-01T01:48:23.493 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-01T01:48:23.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-01T01:48:24.031 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [a,dne1,dne2] 2024-09-01T01:48:24.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-01T01:48:24.586 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [b,dne] 2024-09-01T01:48:24.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-01T01:48:24.957 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-01T01:48:24.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-01T01:48:25.545 INFO:tasks.workunit.client.0.smithi037.stderr:should be safe to stop [dne1,dne2] 2024-09-01T01:48:25.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-01T01:48:25.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:48:25.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:48:25.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:48:25.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:48:25.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:48:25.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:48:25.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:48:25.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:48:25.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:48:25.775 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:48:25.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:48:25.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:48:25.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:48:25.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:48:25.777 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:48:25.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:48:25.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:48:25.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:48:25.778 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:48:25.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:48:25.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:48:25.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:48:25.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:48:25.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:25.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:25.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:48:25.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:48:25.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:48:25.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:48:25.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:48:25.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:48:25.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:48:25.790 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:48:25.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:48:25.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:48:25.791 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:48:25.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:48:25.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:48:25.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:48:25.792 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:48:25.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:48:25.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:48:25.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:48:25.795 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:48:25.795 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:25.795 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:25.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:48:25.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:48:25.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:48:25.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:48:25.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:48:25.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:25.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:25.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:48:25.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:48:25.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:48:25.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:26.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:48:26.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:48:26.427 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:48:26.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:48:26.511 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:48:26.512 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2024-09-01T01:48:26.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:48:27.440 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:48:27.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:48:27.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:27.471 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:48:27.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2024-09-01T01:48:27.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:48:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9bd07327-618a-4bc3-85bf-5a3fe36aa922 2024-09-01T01:48:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9bd07327-618a-4bc3-85bf-5a3fe36aa922' 2024-09-01T01:48:27.473 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 9bd07327-618a-4bc3-85bf-5a3fe36aa922 2024-09-01T01:48:27.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:48:27.488 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDrx9NmYvIWHRAAsGciAB1Z/wsCMJiuI8U3CQ== 2024-09-01T01:48:27.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDrx9NmYvIWHRAAsGciAB1Z/wsCMJiuI8U3CQ=="}' 2024-09-01T01:48:27.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9bd07327-618a-4bc3-85bf-5a3fe36aa922 -i td/ok-to-stop/0/new.json 2024-09-01T01:48:27.760 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:48:27.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2024-09-01T01:48:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrx9NmYvIWHRAAsGciAB1Z/wsCMJiuI8U3CQ== --osd-uuid 9bd07327-618a-4bc3-85bf-5a3fe36aa922 2024-09-01T01:48:27.797 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:27.796+0000 7f30ae8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:27.819 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:27.817+0000 7f30ae8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:27.820 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:27.819+0000 7f30ae8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:27.820 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:27.819+0000 7f30ae8d6640 -1 bdev(0x561237025c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:48:27.820 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:27.819+0000 7f30ae8d6640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-01T01:48:33.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-01T01:48:33.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:48:33.315 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:48:33.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:48:33.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:48:33.908 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:48:33.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:48:33.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:48:33.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:48:33.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:48:33.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:48:33.934 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:33.933+0000 7f57de9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:33.960 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:33.957+0000 7f57de9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:33.961 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:33.958+0000 7f57de9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:48:34.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:48:34.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:48:34.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:48:34.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:34.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:34.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:35.558 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:35.558+0000 7f57de9dd640 -1 Falling back to public interface 2024-09-01T01:48:35.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:35.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:35.764 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:48:35.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:48:35.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:35.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:36.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:37.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:37.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:37.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:48:37.207 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:48:37.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:37.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:37.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:38.294 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:48:38.293+0000 7f57de9dd640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:48:38.657 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:38.657 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:48:38.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:38.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:48:38.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:38.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:39.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:40.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:40.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:40.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:48:40.186 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:48:40.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:40.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:40.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:41.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:41.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:41.610 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:48:41.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:48:41.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:41.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:42.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:43.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:43.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:43.138 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:48:43.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:48:43.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:43.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:43.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:44.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:44.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:44.548 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:48:44.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:48:44.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:44.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:45.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:46.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:46.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:46.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:48:46.030 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:48:46.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:46.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:46.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:47.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:47.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:47.447 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:48:47.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:48:47.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:47.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:47.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:48.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:48.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:48.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:48:48.881 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:48:48.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:48.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:49.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:50.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:50.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:50.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:48:50.353 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:48:50.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:50.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:50.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:51.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:51.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:51.941 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:48:51.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:48:51.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:51.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:52.920 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:53.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:53.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:53.922 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:48:53.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:48:53.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:53.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:54.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:55.598 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:55.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:55.599 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:48:55.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:48:55.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:55.600 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:56.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:57.476 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:48:57.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:48:57.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:57.476 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:58.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:48:59.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:48:59.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:48:59.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:48:59.111 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:48:59.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:48:59.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:48:59.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:00.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:00.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:00.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:49:00.772 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:49:00.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:00.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:49:01.291 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1080736430,v1:127.0.0.1:6803/1080736430] [v2:127.0.0.1:6804/1080736430,v1:127.0.0.1:6805/1080736430] exists,up 9bd07327-618a-4bc3-85bf-5a3fe36aa922 2024-09-01T01:49:01.295 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:49:01.295 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:49:01.295 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:49:01.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:49:01.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2024-09-01T01:49:01.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:49:01.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=01e64ac8-5ad9-48e8-a35c-6e076a685ba7 2024-09-01T01:49:01.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 01e64ac8-5ad9-48e8-a35c-6e076a685ba7' 2024-09-01T01:49:01.299 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 01e64ac8-5ad9-48e8-a35c-6e076a685ba7 2024-09-01T01:49:01.299 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:49:01.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANyNNmeu+LEhAA0gpVeuas6SEjBx7z2xr61g== 2024-09-01T01:49:01.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANyNNmeu+LEhAA0gpVeuas6SEjBx7z2xr61g=="}' 2024-09-01T01:49:01.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 01e64ac8-5ad9-48e8-a35c-6e076a685ba7 -i td/ok-to-stop/1/new.json 2024-09-01T01:49:01.852 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:49:01.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2024-09-01T01:49:01.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANyNNmeu+LEhAA0gpVeuas6SEjBx7z2xr61g== --osd-uuid 01e64ac8-5ad9-48e8-a35c-6e076a685ba7 2024-09-01T01:49:01.891 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:01.891+0000 7f43be558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:01.900 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:01.899+0000 7f43be558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:01.901 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:01.900+0000 7f43be558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:01.901 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:01.901+0000 7f43be558640 -1 bdev(0x5609cfa35c00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2024-09-01T01:49:01.901 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:01.901+0000 7f43be558640 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-01T01:49:07.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-01T01:49:07.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:49:07.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T01:49:07.643 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T01:49:07.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:49:08.204 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T01:49:08.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T01:49:08.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:49:08.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:49:08.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:49:08.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:49:08.229 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:08.229+0000 7fc0c0e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:08.272 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:08.271+0000 7fc0c0e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:08.273 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:08.273+0000 7fc0c0e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:08.636 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:49:08.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:49:08.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:08.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:09.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:09.608+0000 7fc0c0e18640 -1 Falling back to public interface 2024-09-01T01:49:10.054 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:49:10.055 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:49:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:10.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:11.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:11.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:11.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:49:11.506 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:49:11.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:11.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:11.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:12.277 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:12.276+0000 7fc0c0e18640 -1 osd.1 0 log_to_monitors true 2024-09-01T01:49:12.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:12.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:12.985 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:49:12.986 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:49:12.986 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:12.986 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:13.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:14.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:14.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:14.432 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:49:14.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:49:14.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:14.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:14.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:15.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:15.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:15.947 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:49:15.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:49:15.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:15.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:16.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:17.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:17.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:17.391 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:49:17.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:49:17.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:17.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:17.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:18.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:18.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:18.807 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:49:18.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:49:18.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:18.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:19.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:20.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:20.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:20.266 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:49:20.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:49:20.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:20.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:20.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:21.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:21.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:21.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:49:21.859 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:49:21.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:21.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:22.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:23.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:23.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:23.264 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:49:23.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:49:23.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:23.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:23.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:24.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:24.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:24.686 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:49:24.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:49:24.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:24.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:25.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:26.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:26.415 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:49:26.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:49:26.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:26.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:27.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:28.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:28.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:28.222 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:49:28.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:49:28.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:28.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:29.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:30.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:30.315 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:49:30.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:30.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:49:30.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:30.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:30.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:31.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:31.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:31.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:49:31.967 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:49:31.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:31.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:32.458 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:33.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:33.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:33.461 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:49:33.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:49:33.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:33.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:33.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:34.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:34.935 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:49:34.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:34.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:49:34.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:34.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:49:35.440 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/980768700,v1:127.0.0.1:6811/980768700] [v2:127.0.0.1:6812/980768700,v1:127.0.0.1:6813/980768700] exists,up 01e64ac8-5ad9-48e8-a35c-6e076a685ba7 2024-09-01T01:49:35.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:49:35.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:49:35.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:49:35.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-01T01:49:35.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:49:35.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:49:35.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b92b934a-9dfc-4631-aa41-93245bc8c467 2024-09-01T01:49:35.448 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 b92b934a-9dfc-4631-aa41-93245bc8c467 2024-09-01T01:49:35.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b92b934a-9dfc-4631-aa41-93245bc8c467' 2024-09-01T01:49:35.448 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:49:35.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvyNNmpv1tGxAAtObT7CviMPm9g/U9dZmFYA== 2024-09-01T01:49:35.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvyNNmpv1tGxAAtObT7CviMPm9g/U9dZmFYA=="}' 2024-09-01T01:49:35.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b92b934a-9dfc-4631-aa41-93245bc8c467 -i td/ok-to-stop/2/new.json 2024-09-01T01:49:35.921 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:49:35.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2024-09-01T01:49:35.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvyNNmpv1tGxAAtObT7CviMPm9g/U9dZmFYA== --osd-uuid b92b934a-9dfc-4631-aa41-93245bc8c467 2024-09-01T01:49:35.962 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:35.960+0000 7f3aa3efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:35.983 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:35.982+0000 7f3aa3efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:35.984 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:35.983+0000 7f3aa3efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:35.985 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:35.984+0000 7f3aa3efe640 -1 bdev(0x5571bc457c00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2024-09-01T01:49:35.985 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:35.984+0000 7f3aa3efe640 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-01T01:49:41.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-01T01:49:41.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:49:41.972 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-01T01:49:41.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T01:49:41.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:49:42.527 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-01T01:49:42.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T01:49:42.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:49:42.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:49:42.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:49:42.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:49:42.553 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:42.552+0000 7f82a26e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:42.571 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:42.571+0000 7f82a26e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:42.573 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:42.572+0000 7f82a26e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:42.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:49:42.995 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:49:42.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:42.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:43.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:43.919 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:43.919+0000 7f82a26e4640 -1 Falling back to public interface 2024-09-01T01:49:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:44.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:49:44.422 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:49:44.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:44.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:44.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:45.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:45.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:45.856 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:49:45.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:49:45.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:45.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:46.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:46.986 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:49:46.986+0000 7f82a26e4640 -1 osd.2 0 log_to_monitors true 2024-09-01T01:49:47.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:47.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:47.282 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:49:47.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:49:47.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:47.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:47.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:48.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:48.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:48.782 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:49:48.783 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:49:48.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:48.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:49.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:50.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:50.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:50.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:49:50.221 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:49:50.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:50.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:50.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:51.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:49:51.634 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:49:51.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:51.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:52.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:53.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:53.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:53.135 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:49:53.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:49:53.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:53.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:53.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:54.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:54.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:54.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:49:54.583 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:49:54.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:54.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:55.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:56.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:56.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:56.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:49:56.114 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:49:56.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:56.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:56.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:57.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:57.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:57.529 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:49:57.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:49:57.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:57.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:58.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:49:59.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:49:59.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:49:59.010 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:49:59.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:49:59.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:49:59.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:49:59.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:00.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:00.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:00.574 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:50:00.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:50:00.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:00.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:01.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:02.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:02.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:02.343 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:50:02.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:50:02.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:02.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:03.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:04.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:04.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:04.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:50:04.507 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:50:04.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:04.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:05.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:06.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:06.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:06.264 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:50:06.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:50:06.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:06.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:06.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:07.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:07.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:07.996 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:50:07.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:50:07.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:07.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:08.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:09.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:09.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:09.843 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:50:09.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:50:09.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:09.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:50:10.344 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2656241276,v1:127.0.0.1:6819/2656241276] [v2:127.0.0.1:6820/2656241276,v1:127.0.0.1:6821/2656241276] exists,up b92b934a-9dfc-4631-aa41-93245bc8c467 2024-09-01T01:50:10.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:50:10.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:50:10.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-01T01:50:10.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:50:10.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:50:10.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=286cbc20-3031-4bfd-945e-39118b6aff20 2024-09-01T01:50:10.353 INFO:tasks.workunit.client.0.smithi037.stdout:add osd3 286cbc20-3031-4bfd-945e-39118b6aff20 2024-09-01T01:50:10.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 286cbc20-3031-4bfd-945e-39118b6aff20' 2024-09-01T01:50:10.353 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:50:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSyNNmD723FRAAqYrA4PdZudESVaWUKUR9FA== 2024-09-01T01:50:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSyNNmD723FRAAqYrA4PdZudESVaWUKUR9FA=="}' 2024-09-01T01:50:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 286cbc20-3031-4bfd-945e-39118b6aff20 -i td/ok-to-stop/3/new.json 2024-09-01T01:50:10.865 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:50:10.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2024-09-01T01:50:10.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSyNNmD723FRAAqYrA4PdZudESVaWUKUR9FA== --osd-uuid 286cbc20-3031-4bfd-945e-39118b6aff20 2024-09-01T01:50:10.904 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:10.904+0000 7f8d4541b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:10.935 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:10.934+0000 7f8d4541b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:10.935 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:10.935+0000 7f8d4541b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:10.936 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:10.935+0000 7f8d4541b640 -1 bdev(0x564cb1ad5c00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2024-09-01T01:50:10.936 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:10.936+0000 7f8d4541b640 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-01T01:50:17.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-01T01:50:17.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:50:17.309 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd3 key to auth repository 2024-09-01T01:50:17.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-01T01:50:17.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:50:17.879 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.3 2024-09-01T01:50:17.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-01T01:50:17.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:50:17.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:50:17.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:50:17.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:50:17.904 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:17.904+0000 7f8705a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:17.906 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:17.906+0000 7f8705a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:17.907 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:17.907+0000 7f8705a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:18.306 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:50:18.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:50:18.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:18.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:18.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:18.760 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:18.760+0000 7f8705a16640 -1 Falling back to public interface 2024-09-01T01:50:19.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:19.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:19.759 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:50:19.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:50:19.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:19.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:20.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:21.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:21.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:21.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:50:21.203 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:50:21.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:21.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:21.446 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:50:21.445+0000 7f8705a16640 -1 osd.3 0 log_to_monitors true 2024-09-01T01:50:21.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:50:22.647 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:50:22.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:22.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:23.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:24.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:24.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:24.253 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:50:24.254 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:50:24.254 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:24.254 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:24.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:25.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:25.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:25.830 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:50:25.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:50:25.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:25.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:26.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:27.314 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:50:27.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:27.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:50:27.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:27.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:27.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:28.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:28.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:28.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:50:28.858 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:50:28.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:28.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:29.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:30.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:30.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:30.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:50:30.293 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:50:30.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:30.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:30.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:31.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:31.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:31.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:50:31.805 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:50:31.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:31.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:32.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:33.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:33.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:33.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:50:33.278 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:50:33.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:33.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:33.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:34.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:50:34.758 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:50:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:34.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:35.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:36.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:36.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:36.745 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:50:36.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:50:36.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:36.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:37.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:38.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:38.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:38.615 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:50:38.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:50:38.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:38.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:39.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:40.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:40.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:40.571 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:50:40.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:50:40.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:40.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:42.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:42.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:42.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:50:42.307 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:50:42.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:42.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:43.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:44.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:50:44.342 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:50:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:44.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:44.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:50:45.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:50:45.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:50:45.888 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:50:45.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:50:45.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:50:45.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T01:50:46.350 INFO:tasks.workunit.client.0.smithi037.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/642394725,v1:127.0.0.1:6827/642394725] [v2:127.0.0.1:6828/642394725,v1:127.0.0.1:6829/642394725] exists,up 286cbc20-3031-4bfd-945e-39118b6aff20 2024-09-01T01:50:46.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:50:46.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:50:46.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:50:46.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-01T01:50:46.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-01T01:50:47.811 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'ec' already exists 2024-09-01T01:50:47.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-01T01:50:47.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T01:50:47.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T01:50:47.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T01:50:47.823 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T01:50:47.824 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T01:50:47.824 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T01:50:47.824 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T01:50:47.824 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T01:50:47.824 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:50:47.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:50:48.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr:2 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr:3' 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:48.696 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:50:48.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-01T01:50:48.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-01T01:50:48.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-01T01:50:48.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:48.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:50:49.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-01T01:50:49.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-01T01:50:49.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672980' 2024-09-01T01:50:49.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:49.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:50:49.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-01T01:50:49.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-01T01:50:49.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672980 2-64424509455' 2024-09-01T01:50:49.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:49.118 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T01:50:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345925 2024-09-01T01:50:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345925 2024-09-01T01:50:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672980 2-64424509455 3-85899345925' 2024-09-01T01:50:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:49.252 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-01T01:50:49.252 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:49.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:50:49.252 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-01T01:50:49.252 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:49.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-01T01:50:49.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-01T01:50:49.254 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836507 2024-09-01T01:50:49.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:50:49.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-01T01:50:49.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:50:50.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:50:50.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:50:51.131 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-01T01:50:51.131 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:51.132 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-01T01:50:51.132 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:51.132 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:50:51.133 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-01T01:50:51.133 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:51.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-01T01:50:51.134 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672980 2024-09-01T01:50:51.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-01T01:50:51.135 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:50:51.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-01T01:50:51.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:51.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-01T01:50:51.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:51.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:50:51.568 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-01T01:50:51.569 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:51.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-01T01:50:51.570 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509455 2024-09-01T01:50:51.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-01T01:50:51.570 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:50:52.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-01T01:50:52.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:52.001 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345925 2024-09-01T01:50:52.001 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:52.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T01:50:52.002 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345925 2024-09-01T01:50:52.002 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:52.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345925 2024-09-01T01:50:52.003 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.3 seq 85899345925 2024-09-01T01:50:52.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-09-01T01:50:52.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T01:50:52.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345925 2024-09-01T01:50:52.427 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T01:50:52.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:50:52.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:50:52.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:50:53.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=24 2024-09-01T01:50:53.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:50:53.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:50:53.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 24 = 32 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 24 '!=' -1 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=24 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T01:50:53.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T01:50:54.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T01:50:54.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:50:54.012 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:50:54.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-01T01:50:54.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:50:54.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:50:54.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:50:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-01T01:50:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T01:50:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T01:50:54.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-01T01:50:55.547 INFO:tasks.workunit.client.0.smithi037.stderr:set pool 1 min_size to 3 2024-09-01T01:50:55.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-01T01:50:55.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T01:50:55.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:50:55.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:2 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:3' 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:56.105 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:50:56.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-01T01:50:56.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-01T01:50:56.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-01T01:50:56.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:56.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:50:56.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-01T01:50:56.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-01T01:50:56.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984' 2024-09-01T01:50:56.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:56.330 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:50:56.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-01T01:50:56.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-01T01:50:56.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509459' 2024-09-01T01:50:56.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:50:56.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345929 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345929 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509459 3-85899345929' 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:56.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:50:56.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-01T01:50:56.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:56.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-01T01:50:56.556 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836510 2024-09-01T01:50:56.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-01T01:50:56.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:50:57.014 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-01T01:50:57.014 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:50:58.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:50:58.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:50:58.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-01T01:50:58.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:58.467 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-01T01:50:58.468 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:58.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:50:58.468 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-01T01:50:58.469 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:58.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-01T01:50:58.470 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672984 2024-09-01T01:50:58.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-01T01:50:58.470 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:50:58.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-01T01:50:58.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:58.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-01T01:50:58.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:58.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:50:58.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-01T01:50:58.864 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:58.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-01T01:50:58.865 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509459 2024-09-01T01:50:58.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-01T01:50:58.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:50:59.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-01T01:50:59.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:50:59.286 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345929 2024-09-01T01:50:59.286 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:50:59.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T01:50:59.287 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345929 2024-09-01T01:50:59.287 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:50:59.288 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345929 2024-09-01T01:50:59.288 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.3 seq 85899345929 2024-09-01T01:50:59.289 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-09-01T01:50:59.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T01:50:59.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345929 -lt 85899345929 2024-09-01T01:50:59.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T01:50:59.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:50:59.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:00.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T01:51:00.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:51:00.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:51:00.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-01T01:51:00.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:51:00.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:00.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:01.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-01T01:51:01.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T01:51:01.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T01:51:01.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-01T01:51:01.593 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:01.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-01T01:51:01.992 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:02.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-01T01:51:02.393 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:02.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-01T01:51:02.786 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:02.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-01T01:51:03.070 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:03.070 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:03.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-01T01:51:03.346 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:03.346 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:03.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-01T01:51:03.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-01T01:51:03.749 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:03.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-01T01:51:03.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-01T01:51:04.143 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:04.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-01T01:51:04.908 INFO:tasks.workunit.client.0.smithi037.stderr:set pool 1 min_size to 2 2024-09-01T01:51:04.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T01:51:04.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:51:05.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:51:05.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:51:05.533 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:51:05.533 INFO:tasks.workunit.client.0.smithi037.stderr:2 2024-09-01T01:51:05.533 INFO:tasks.workunit.client.0.smithi037.stderr:3' 2024-09-01T01:51:05.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:51:05.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:05.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:51:05.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-01T01:51:05.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-01T01:51:05.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-01T01:51:05.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:05.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:51:05.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-01T01:51:05.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-01T01:51:05.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988' 2024-09-01T01:51:05.869 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:05.870 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:51:05.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-01T01:51:05.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-01T01:51:05.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988 2-64424509463' 2024-09-01T01:51:05.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:05.995 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T01:51:06.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345933 2024-09-01T01:51:06.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345933 2024-09-01T01:51:06.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988 2-64424509463 3-85899345933' 2024-09-01T01:51:06.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:06.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-01T01:51:06.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:06.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:51:06.109 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-01T01:51:06.109 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:06.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-01T01:51:06.110 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836514 2024-09-01T01:51:06.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-01T01:51:06.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:51:06.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836514 2024-09-01T01:51:06.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:51:07.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:51:07.523 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:51:07.976 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836514 2024-09-01T01:51:07.976 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:07.976 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-01T01:51:07.976 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:07.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:51:07.977 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-01T01:51:07.977 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:07.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-01T01:51:07.979 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672988 2024-09-01T01:51:07.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-01T01:51:07.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:51:08.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-01T01:51:08.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:08.403 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-01T01:51:08.403 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:08.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:51:08.404 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-01T01:51:08.404 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:08.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-01T01:51:08.405 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509463 2024-09-01T01:51:08.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-01T01:51:08.405 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:51:08.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-01T01:51:08.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:08.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345933 2024-09-01T01:51:08.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:08.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T01:51:08.838 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345933 2024-09-01T01:51:08.838 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:08.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345933 2024-09-01T01:51:08.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-09-01T01:51:08.839 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.3 seq 85899345933 2024-09-01T01:51:08.839 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T01:51:09.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345933 -lt 85899345933 2024-09-01T01:51:09.263 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T01:51:09.263 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:09.263 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:09.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T01:51:09.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:51:09.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:51:09.808 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:51:09.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:51:09.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:51:09.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:51:09.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:51:10.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=12 2024-09-01T01:51:10.228 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:51:10.228 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:10.228 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:10.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 12 = 32 2024-09-01T01:51:10.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 12 '!=' -1 2024-09-01T01:51:10.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T01:51:10.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=12 2024-09-01T01:51:10.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T01:51:10.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:51:10.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:51:11.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=12 2024-09-01T01:51:11.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:51:11.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:11.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 12 = 32 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 12 '!=' 12 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T01:51:11.862 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T01:51:11.863 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T01:51:12.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T01:51:12.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T01:51:12.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-01T01:51:12.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T01:51:12.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T01:51:12.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T01:51:12.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T01:51:12.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T01:51:12.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T01:51:12.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T01:51:12.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T01:51:12.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T01:51:12.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T01:51:12.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-01T01:51:12.995 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T01:51:12.995 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:12.995 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:13.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-01T01:51:13.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T01:51:13.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T01:51:13.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-01T01:51:14.018 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-01T01:51:14.416 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:14.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-01T01:51:14.701 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:14.701 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:14.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-01T01:51:14.979 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:14.979 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:14.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-01T01:51:14.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-01T01:51:15.382 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:15.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-01T01:51:15.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-01T01:51:15.784 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:15.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-01T01:51:15.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-01T01:51:16.186 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:16.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-01T01:51:16.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-01T01:51:16.584 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:16.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-01T01:51:16.585 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:16.585 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:16.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:16.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:16.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:16.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:16.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-01T01:51:17.693 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already down. 2024-09-01T01:51:17.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-01T01:51:17.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1690: wait_for_peered: local cmd= 2024-09-01T01:51:17.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1691: wait_for_peered: local num_peered=-1 2024-09-01T01:51:17.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1692: wait_for_peered: local cur_peered 2024-09-01T01:51:17.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: get_timeout_delays 90 .1 2024-09-01T01:51:17.706 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T01:51:17.706 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T01:51:17.706 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T01:51:17.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T01:51:17.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: wait_for_peered: local -a delays 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1694: wait_for_peered: local -i loop=0 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1696: wait_for_peered: flush_pg_stats 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:51:17.792 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:51:18.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr:2 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr:3' 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:18.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:51:18.420 INFO:tasks.workunit.client.0.smithi037.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-01T01:51:18.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T01:51:18.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T01:51:18.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T01:51:18.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:18.423 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T01:51:18.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672992 2024-09-01T01:51:18.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672992 2024-09-01T01:51:18.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672992' 2024-09-01T01:51:18.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:18.579 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T01:51:18.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-01T01:51:18.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-01T01:51:18.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672992 2-64424509467' 2024-09-01T01:51:18.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:51:18.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T01:51:18.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345937 2024-09-01T01:51:18.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345937 2024-09-01T01:51:18.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672992 2-64424509467 3-85899345937' 2024-09-01T01:51:18.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:18.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672992 2024-09-01T01:51:18.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:18.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T01:51:18.843 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672992 2024-09-01T01:51:18.843 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:18.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672992 2024-09-01T01:51:18.844 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672992 2024-09-01T01:51:18.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-01T01:51:18.845 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:51:19.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672992 2024-09-01T01:51:19.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:51:20.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:51:20.282 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T01:51:20.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-01T01:51:20.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:20.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-01T01:51:20.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:20.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T01:51:20.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-01T01:51:20.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:20.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-01T01:51:20.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-01T01:51:20.706 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.2 seq 64424509467 2024-09-01T01:51:20.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T01:51:21.131 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-01T01:51:21.131 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:51:21.131 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345937 2024-09-01T01:51:21.131 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:51:21.132 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T01:51:21.132 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345937 2024-09-01T01:51:21.132 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:51:21.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345937 2024-09-01T01:51:21.133 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.3 seq 85899345937 2024-09-01T01:51:21.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-01T01:51:21.134 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T01:51:21.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-01T01:51:21.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: wait_for_peered: get_num_pgs 2024-09-01T01:51:21.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:21.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: wait_for_peered: test 32 == 0 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: true 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: get_num_active_or_peered 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-01T01:51:22.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-01T01:51:22.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-01T01:51:22.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-01T01:51:22.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: cur_peered=32 2024-09-01T01:51:22.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-01T01:51:22.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T01:51:22.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T01:51:23.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 = 32 2024-09-01T01:51:23.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: break 2024-09-01T01:51:23.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1721: wait_for_peered: return 0 2024-09-01T01:51:23.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-01T01:51:23.451 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-01T01:51:23.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-01T01:51:23.845 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:23.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-01T01:51:24.132 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:24.132 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:24.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-01T01:51:24.410 INFO:tasks.workunit.client.0.smithi037.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-01T01:51:24.411 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-01T01:51:24.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:24.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:24.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:24.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:24.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:24.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:24.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:24.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:24.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:24.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:24.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:24.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:24.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:24.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:24.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:24.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:24.532 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:24.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:24.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:24.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:24.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:24.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:24.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:24.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:24.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:24.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:24.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:24.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:24.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:24.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:24.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:51:24.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:24.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:24.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:24.561 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:24.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:24.562 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:24.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:24.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:24.564 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:24.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:24.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:24.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:24.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:24.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:24.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:24.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:24.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:24.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:24.568 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:24.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:24.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:24.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:24.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:51:24.570 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:51:24.570 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:24.570 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-01T01:51:24.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:51:24.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:51:25.034 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:25.034 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:25.034 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:25.034 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:25.035 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:25.035 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:25.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:25.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:51:25.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:25.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:51:25.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:25.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:25.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:25.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:51:25.294 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:25.295 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:25.296 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:25.296 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:51:25.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-01T01:51:26.166 INFO:tasks.workunit.client.0.smithi037.stderr:invalid mon(s) specified: dne 2024-09-01T01:51:26.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-01T01:51:26.337 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a 2024-09-01T01:51:26.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-01T01:51:26.485 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-01T01:51:26.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-01T01:51:26.636 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-01T01:51:26.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-01T01:51:26.829 INFO:tasks.workunit.client.0.smithi037.stderr:mon.dne does not exist 2024-09-01T01:51:26.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:26.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:26.840 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:26.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:26.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:26.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:26.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:26.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:26.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:26.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:26.949 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:26.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:26.950 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:26.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:26.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:26.951 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:26.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:26.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:26.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:26.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:26.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:26.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:26.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:26.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:26.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:26.956 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:26.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:26.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:26.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:26.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:26.959 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:26.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:26.960 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:26.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:26.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:26.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:26.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:26.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:26.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:26.962 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:26.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:26.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:26.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:26.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:26.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:26.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:26.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:26.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:26.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:26.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:51:26.967 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:51:26.967 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:26.967 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:26.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:51:26.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:51:26.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:51:26.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:51:26.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:51:26.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:51:26.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:27.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:27.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:51:27.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:27.619 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:51:27.619 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:27.620 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:27.620 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:27.620 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:27.621 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:27.622 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:27.622 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:51:27.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:51:27.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:27.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:27.708 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:27.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:27.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:51:27.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-01T01:51:27.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:27.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:27.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-01T01:51:27.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:27.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-01T01:51:27.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-01T01:51:28.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:28.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:28.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:28.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:28.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-01T01:51:28.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:28.406 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:51:28.407 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:28.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get fsid 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:51:28.490 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:28.491 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get mon_host 2024-09-01T01:51:28.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-01T01:51:31.765 INFO:tasks.workunit.client.0.smithi037.stderr:invalid mon(s) specified: dne 2024-09-01T01:51:31.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:51:31.953 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons a 2024-09-01T01:51:31.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:51:32.152 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b 2024-09-01T01:51:32.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-01T01:51:32.320 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b 2024-09-01T01:51:32.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:51:32.562 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:51:32.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:51:32.786 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:51:32.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:51:33.050 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:51:33.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-01T01:51:33.302 INFO:tasks.workunit.client.0.smithi037.stderr:mon.dne does not exist 2024-09-01T01:51:33.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:33.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:33.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:33.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:33.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:33.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:33.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:33.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:33.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:33.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:33.421 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:33.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:33.422 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:33.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:33.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:33.423 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:33.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:33.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:33.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:33.425 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:33.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:33.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:33.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:33.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:33.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:33.428 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:33.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:51:33.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:33.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:33.432 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:51:33.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:51:33.433 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:51:33.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:51:33.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:51:33.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:51:33.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:33.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:51:33.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:51:33.435 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:51:33.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:51:33.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:51:33.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:51:33.438 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:51:33.438 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:33.438 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:33.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:51:33.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:51:33.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:51:33.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:51:33.440 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:51:33.440 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:33.440 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:33.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-01T01:51:33.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:51:33.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:51:33.916 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:33.917 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:51:34.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:34.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:51:34.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:34.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:34.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:34.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:34.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:51:34.210 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:34.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:51:34.289 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-01T01:51:34.289 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:34.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:34.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-01T01:51:34.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:34.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-01T01:51:34.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-01T01:51:34.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:34.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-01T01:51:34.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:51:34.947 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:51:34.948 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:51:34.948 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:34.948 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:34.948 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:34.948 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:51:34.949 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:34.949 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get fsid 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:51:35.031 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:35.032 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get mon_host 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-01T01:51:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-01T01:51:35.548 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:51:35.549 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-01T01:51:35.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:51:35.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-01T01:51:35.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:35.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:51:35.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:51:35.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:51:35.728 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:51:35.728 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:51:35.728 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:51:35.728 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:35.729 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:35.729 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:35.729 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:51:35.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:35.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get fsid 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:51:35.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:51:35.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get mon_host 2024-09-01T01:51:35.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-01T01:51:35.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:51:35.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-01T01:51:35.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:51:35.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-01T01:51:35.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:51:35.891 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:51:35.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155555 2024-09-01T01:51:35.891 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:35.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155495 -lt 1725155555 ]] 2024-09-01T01:51:35.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:51:35.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:39.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:39.098 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:39.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:39.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:51:39.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:51:39.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:51:39.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:51:39.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:51:39.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:51:39.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-01T01:51:39.402 INFO:tasks.workunit.client.0.smithi037.stderr:invalid mon(s) specified: dne 2024-09-01T01:51:39.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:51:39.669 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c) after stopping a 2024-09-01T01:51:39.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:51:39.945 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,c) after stopping b 2024-09-01T01:51:39.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:51:40.222 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b) after stopping c 2024-09-01T01:51:40.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-01T01:51:40.421 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a b 2024-09-01T01:51:40.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-01T01:51:40.632 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b c 2024-09-01T01:51:40.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-01T01:51:40.874 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b c 2024-09-01T01:51:40.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:51:41.177 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:51:41.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:51:41.445 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:51:41.454 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:51:41.713 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:51:41.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:51:41.990 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.c 2024-09-01T01:51:41.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-01T01:51:41.999 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:51:41.999 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:51:42.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:51:42.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:51:42.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:51:42.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:51:42.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-01T01:51:42.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:51:42.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2024-09-01T01:51:42.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:51:42.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2024-09-01T01:51:42.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:51:42.106 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:51:42.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155562 2024-09-01T01:51:42.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:42.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155502 -lt 1725155562 ]] 2024-09-01T01:51:42.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:42.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:42.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:42.302 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.303 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.303 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:42.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:42.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:42.305 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:42.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155502 -lt 1725155562 ]] 2024-09-01T01:51:42.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:42.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:42.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:42.501 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.501 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.502 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:42.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:42.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:42.504 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:42.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155502 -lt 1725155562 ]] 2024-09-01T01:51:42.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:42.505 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:42.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:42.700 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.701 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:42.701 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:42.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:42.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:42.703 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:42.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155502 -lt 1725155562 ]] 2024-09-01T01:51:42.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:42.704 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:45.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:45.909 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:45.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:45.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:45.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:45.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:45.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:45.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155505 -lt 1725155562 ]] 2024-09-01T01:51:45.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:45.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:46.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:46.108 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:46.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:46.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:46.111 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:46.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155506 -lt 1725155562 ]] 2024-09-01T01:51:46.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:46.112 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:46.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:46.306 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:46.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:46.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:46.309 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:46.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155506 -lt 1725155562 ]] 2024-09-01T01:51:46.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:46.310 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:46.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:46.508 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.509 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.509 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:46.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:46.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:46.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:46.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155506 -lt 1725155562 ]] 2024-09-01T01:51:46.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:46.513 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:46.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:46.712 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:46.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155506 -lt 1725155562 ]] 2024-09-01T01:51:46.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:46.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:46.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:46.908 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:46.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:46.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:46.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:46.911 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:46.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155506 -lt 1725155562 ]] 2024-09-01T01:51:46.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:46.912 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:47.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:47.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:47.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:47.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:47.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155507 -lt 1725155562 ]] 2024-09-01T01:51:47.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:47.111 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:47.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:47.308 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:47.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:47.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:47.310 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:47.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155507 -lt 1725155562 ]] 2024-09-01T01:51:47.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:47.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:47.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:47.506 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.507 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.507 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:47.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:47.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:47.509 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:47.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155507 -lt 1725155562 ]] 2024-09-01T01:51:47.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:47.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:47.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:47.706 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:47.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:47.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:47.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:47.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155507 -lt 1725155562 ]] 2024-09-01T01:51:47.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:47.710 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:47.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:47.905 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:47.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:47.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:47.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:47.908 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:47.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155507 -lt 1725155562 ]] 2024-09-01T01:51:47.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:47.909 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:51.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:51.113 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:51.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:51.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:51.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:51:51.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:51:51.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:51:51.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155511 -lt 1725155562 ]] 2024-09-01T01:51:51.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-01T01:51:51.117 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:51:57.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:51:57.472 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:57.472 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:51:33.462422Z","created":"2024-09-01T01:51:33.462422Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:51:57.472 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2024-09-01T01:51:57.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:51:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:51:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:51:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:51:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:51:57.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:51:57.809 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a 2024-09-01T01:51:57.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:51:58.036 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,c) after stopping b 2024-09-01T01:51:58.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:51:58.236 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons c 2024-09-01T01:51:58.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:51:58.445 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-01T01:51:58.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:52:01.598 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-01T01:52:01.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:52:01.871 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:52:01.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:52:02.076 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-01T01:52:02.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:02.079 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:02.079 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:02.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:02.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:02.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:02.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:02.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:52:02.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:52:02.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:52:02.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:02.186 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:02.186 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:02.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:02.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:02.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:02.189 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:02.189 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:52:02.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:52:02.190 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:52:02.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:52:02.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:52:02.191 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:52:02.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:02.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:52:02.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:52:02.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:02.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:52:02.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:52:02.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:52:02.198 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:52:02.198 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.198 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:52:02.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:52:02.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:52:02.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:02.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:02.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:02.202 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:52:02.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:52:02.202 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:52:02.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:52:02.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:52:02.204 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:52:02.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:02.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:52:02.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:52:02.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:02.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:52:02.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:52:02.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:52:02.207 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:52:02.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:52:02.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:52:02.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-01T01:52:02.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:52:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:52:02.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:02.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:02.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:52:02.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:52:02.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:52:02.884 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:02.885 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.885 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.885 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:52:02.885 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:02.885 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:52:02.970 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:02.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:52:03.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-01T01:52:03.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-01T01:52:03.517 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:03.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-01T01:52:03.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:03.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-01T01:52:03.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:03.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:52:03.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:03.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:03.721 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:03.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get fsid 2024-09-01T01:52:03.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:03.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:52:03.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:03.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get mon_host 2024-09-01T01:52:03.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-01T01:52:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:04.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-01T01:52:04.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:04.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-01T01:52:04.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:04.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:52:04.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:04.551 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:04.551 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:52:04.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:04.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get fsid 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:52:04.636 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:04.637 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get mon_host 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2024-09-01T01:52:04.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:05.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-01T01:52:05.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:05.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-01T01:52:05.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:05.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-01T01:52:05.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:05.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:05.371 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:05.372 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.d.asok 2024-09-01T01:52:05.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:05.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.d.asok config get fsid 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:05.460 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.d.asok 2024-09-01T01:52:05.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:05.460 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.d.asok config get mon_host 2024-09-01T01:52:05.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-01T01:52:05.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155585 2024-09-01T01:52:05.543 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:05.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155525 -lt 1725155585 ]] 2024-09-01T01:52:05.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:52:05.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:08.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:08.750 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:08.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:08.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:52:08.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:52:08.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:52:08.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:52:08.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:52:08.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:52:08.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-01T01:52:09.073 INFO:tasks.workunit.client.0.smithi037.stderr:invalid mon(s) specified: dne 2024-09-01T01:52:09.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:52:12.277 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-01T01:52:12.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:52:12.550 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,c,d) after stopping b 2024-09-01T01:52:12.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:52:12.809 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,d) after stopping c 2024-09-01T01:52:12.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-01T01:52:13.061 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,c) after stopping d 2024-09-01T01:52:13.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-01T01:52:13.295 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons a b 2024-09-01T01:52:13.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-01T01:52:13.460 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons c d 2024-09-01T01:52:13.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:52:13.757 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:52:13.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:52:14.091 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:52:14.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:52:14.411 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:52:14.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:52:14.648 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.c 2024-09-01T01:52:14.657 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-01T01:52:14.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:14.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:14.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:14.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:14.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:14.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:14.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-01T01:52:14.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:52:14.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155594 2024-09-01T01:52:14.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:14.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155534 -lt 1725155594 ]] 2024-09-01T01:52:14.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:14.765 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:14.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:14.961 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:14.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:14.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:14.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:14.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:14.964 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:14.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155534 -lt 1725155594 ]] 2024-09-01T01:52:14.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:14.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:15.160 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:15.160 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:15.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:15.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:15.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:15.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155535 -lt 1725155594 ]] 2024-09-01T01:52:15.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:15.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:15.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:15.358 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:15.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:15.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:15.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:15.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155535 -lt 1725155594 ]] 2024-09-01T01:52:15.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:15.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:15.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:15.559 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:15.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:15.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:15.562 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:15.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155535 -lt 1725155594 ]] 2024-09-01T01:52:15.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:15.563 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:15.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:15.759 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.759 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.759 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:15.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155535 -lt 1725155594 ]] 2024-09-01T01:52:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:15.763 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:15.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:15.960 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:15.961 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:15.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:15.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:15.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:15.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155535 -lt 1725155594 ]] 2024-09-01T01:52:15.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:15.964 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:16.160 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:16.160 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:16.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:16.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:16.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:16.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155536 -lt 1725155594 ]] 2024-09-01T01:52:16.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:16.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:16.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:16.361 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:16.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:16.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:16.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:16.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155536 -lt 1725155594 ]] 2024-09-01T01:52:16.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:16.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:16.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:16.561 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.565 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:16.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:16.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:16.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:16.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155536 -lt 1725155594 ]] 2024-09-01T01:52:16.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:16.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:16.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:16.761 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.761 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.761 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:16.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155536 -lt 1725155594 ]] 2024-09-01T01:52:16.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:16.765 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:16.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:16.959 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.960 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:16.960 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:16.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:16.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:16.962 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:16.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155536 -lt 1725155594 ]] 2024-09-01T01:52:16.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:16.963 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:17.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:17.161 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:17.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:17.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:17.164 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:17.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155537 -lt 1725155594 ]] 2024-09-01T01:52:17.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:17.165 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:17.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:17.359 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:17.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:17.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:17.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:17.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155537 -lt 1725155594 ]] 2024-09-01T01:52:17.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:17.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:17.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:17.557 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.557 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.558 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:17.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:17.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:17.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:17.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155537 -lt 1725155594 ]] 2024-09-01T01:52:17.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:17.561 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:17.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:17.756 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:17.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:17.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:17.759 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:17.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155537 -lt 1725155594 ]] 2024-09-01T01:52:17.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:17.760 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:17.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:17.954 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:17.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:17.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:17.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:17.957 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:17.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155537 -lt 1725155594 ]] 2024-09-01T01:52:17.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:17.958 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:18.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:18.154 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:18.155 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:18.155 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:18.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:18.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:18.157 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:18.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155538 -lt 1725155594 ]] 2024-09-01T01:52:18.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:18.158 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:21.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:21.360 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:21.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:21.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:21.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:21.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:21.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:21.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155541 -lt 1725155594 ]] 2024-09-01T01:52:21.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:21.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:21.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:21.559 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:21.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:21.560 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:21.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:52:21.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:52:21.563 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:21.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155541 -lt 1725155594 ]] 2024-09-01T01:52:21.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:52:21.564 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:34.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:34.895 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:34.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:02.232437Z","created":"2024-09-01T01:52:02.232437Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:34.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:52:34.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:52:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-01T01:52:35.400 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:52:35.659 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-09-01T01:52:35.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:52:38.816 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-09-01T01:52:38.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-01T01:52:39.028 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-09-01T01:52:39.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:52:42.225 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:52:42.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:52:42.482 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:52:42.491 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:52:42.717 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:52:42.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:52:42.969 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.c 2024-09-01T01:52:42.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-01T01:52:43.227 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.d 2024-09-01T01:52:43.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:43.237 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:43.237 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:43.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:43.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:43.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:43.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:43.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:52:43.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:43.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:43.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:43.349 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:52:43.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:52:43.350 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:52:43.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:52:43.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:52:43.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:52:43.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:43.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:52:43.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:52:43.352 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:43.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:52:43.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:52:43.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:52:43.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:52:43.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:43.360 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:43.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:52:43.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:52:43.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:52:43.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:52:43.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:52:43.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:52:43.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:52:43.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:52:43.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:52:43.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:52:43.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:52:43.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:43.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:52:43.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:52:43.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:52:43.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:52:43.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:52:43.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:52:43.369 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:52:43.369 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:43.369 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:43.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:52:43.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:52:43.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:52:43.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2024-09-01T01:52:43.371 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:52:43.371 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:43.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:43.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.42777 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=aa51ec55-776e-43bf-ab48-832ca942c410 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:43.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:52:43.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:43.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2024-09-01T01:52:43.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:43.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:43.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:43.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-01T01:52:44.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:52:44.011 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:52:44.012 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:44.013 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get fsid 2024-09-01T01:52:44.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:52:44.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:44.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:52:44.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.a.asok 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:44.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.a.asok config get mon_host 2024-09-01T01:52:44.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-01T01:52:44.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:44.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:44.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2024-09-01T01:52:44.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:44.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2024-09-01T01:52:44.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:44.646 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:44.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:44.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-01T01:52:44.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:44.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2024-09-01T01:52:44.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:44.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:52:44.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:44.831 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:44.832 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:52:44.832 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:52:44.832 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:52:44.832 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:44.833 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:44.833 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:44.833 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:52:44.833 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:44.833 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get fsid 2024-09-01T01:52:44.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.b.asok 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:44.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.b.asok config get mon_host 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2024-09-01T01:52:44.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:45.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-01T01:52:45.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:45.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2024-09-01T01:52:45.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:45.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:52:45.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:45.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:45.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:52:45.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:52:45.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:52:45.611 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:45.612 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:45.612 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:45.612 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:52:45.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:45.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get fsid 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2024-09-01T01:52:45.694 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.c.asok 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:45.695 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.c.asok config get mon_host 2024-09-01T01:52:45.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2024-09-01T01:52:45.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-01T01:52:46.195 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:46.195 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:46.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-01T01:52:46.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-01T01:52:46.415 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.d.asok 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:46.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.d.asok config get fsid 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:46.496 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:46.497 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:46.497 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.d.asok 2024-09-01T01:52:46.497 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:46.497 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.d.asok config get mon_host 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2024-09-01T01:52:46.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.42777/$cluster-$name.asok' 2024-09-01T01:52:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.42777/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-01T01:52:47.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:52:47.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.e.asok 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:47.254 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.e.asok config get fsid 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:52:47.335 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.42777/ceph-mon.e.asok 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:52:47.336 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.42777/ceph-mon.e.asok config get mon_host 2024-09-01T01:52:47.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:52:47.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:52:47.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155627 2024-09-01T01:52:47.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:52:47.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155567 -lt 1725155627 ]] 2024-09-01T01:52:47.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-01T01:52:47.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:52:53.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:52:53.624 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:53.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:52:53.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:52:53.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-01T01:52:53.889 INFO:tasks.workunit.client.0.smithi037.stderr:invalid mon(s) specified: dne 2024-09-01T01:52:53.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:52:54.191 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-09-01T01:52:54.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:52:54.494 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,c,d,e) after stopping b 2024-09-01T01:52:54.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:52:54.761 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,d,e) after stopping c 2024-09-01T01:52:54.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-01T01:52:55.048 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,c,e) after stopping d 2024-09-01T01:52:55.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-01T01:52:55.331 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,c,d) after stopping e 2024-09-01T01:52:55.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-01T01:52:55.617 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (c,d,e) after stopping a b 2024-09-01T01:52:55.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-01T01:52:58.823 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (a,b,e) after stopping c d 2024-09-01T01:52:58.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-01T01:52:59.100 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons a b c 2024-09-01T01:52:59.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:52:59.289 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:52:59.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:52:59.560 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:52:59.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:53:02.763 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:53:02.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:53:03.037 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.c 2024-09-01T01:53:03.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-01T01:53:03.315 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.d 2024-09-01T01:53:03.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-01T01:53:03.592 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.e 2024-09-01T01:53:03.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-01T01:53:03.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:53:03.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:53:03.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:53:03.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:53:03.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:53:03.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:53:03.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-01T01:53:03.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:53:03.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2024-09-01T01:53:03.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:53:03.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2024-09-01T01:53:03.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:53:03.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:53:03.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155643 2024-09-01T01:53:03.709 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:03.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155583 -lt 1725155643 ]] 2024-09-01T01:53:03.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:53:03.710 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:06.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:06.912 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:06.912 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:53:06.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:06.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:06.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:06.915 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:06.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155586 -lt 1725155643 ]] 2024-09-01T01:53:06.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:53:06.916 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:10.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:10.121 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:10.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:10.122 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:53:10.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:10.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:10.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:10.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155590 -lt 1725155643 ]] 2024-09-01T01:53:10.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:53:10.125 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:10.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:10.322 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:10.323 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:53:10.323 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:10.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:10.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:10.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:10.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155590 -lt 1725155643 ]] 2024-09-01T01:53:10.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:53:10.326 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:13.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:13.529 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:13.529 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:13.529 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:53:13.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:13.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:13.532 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:13.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155593 -lt 1725155643 ]] 2024-09-01T01:53:13.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-01T01:53:13.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:25.739 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:25.739 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:53:25.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:53:26.048 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d,e) after stopping a 2024-09-01T01:53:26.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:53:26.380 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (c,d,e) after stopping b 2024-09-01T01:53:26.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:53:26.659 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,d,e) after stopping c 2024-09-01T01:53:26.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-01T01:53:26.944 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,e) after stopping d 2024-09-01T01:53:26.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-01T01:53:27.247 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d) after stopping e 2024-09-01T01:53:27.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:53:27.473 INFO:tasks.workunit.client.0.smithi037.stderr:adding another mon that is not yet online will not break quorum 2024-09-01T01:53:27.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:53:27.718 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:53:27.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:53:28.010 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.b 2024-09-01T01:53:28.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:53:28.280 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.c 2024-09-01T01:53:28.289 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-01T01:53:31.484 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.d 2024-09-01T01:53:31.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-01T01:53:31.799 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.e 2024-09-01T01:53:31.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-01T01:53:31.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:53:31.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:53:31.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:53:31.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:53:31.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2024-09-01T01:53:31.915 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2024-09-01T01:53:31.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1725155671 2024-09-01T01:53:31.916 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:31.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155611 -lt 1725155671 ]] 2024-09-01T01:53:31.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:31.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:32.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:32.112 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:32.112 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:32.112 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:32.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:32.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:32.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:32.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155612 -lt 1725155671 ]] 2024-09-01T01:53:32.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:32.117 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:35.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:35.321 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:35.321 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:35.321 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:35.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:35.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:35.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:35.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155615 -lt 1725155671 ]] 2024-09-01T01:53:35.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:35.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:35.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:35.521 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:35.521 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:35.521 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:35.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:35.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:35.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:35.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155615 -lt 1725155671 ]] 2024-09-01T01:53:35.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:35.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:38.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:38.727 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:38.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:38.728 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:38.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:38.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:38.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:38.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155618 -lt 1725155671 ]] 2024-09-01T01:53:38.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:38.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:38.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:38.927 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:38.928 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:38.928 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:38.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:38.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:38.930 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:38.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155618 -lt 1725155671 ]] 2024-09-01T01:53:38.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:38.931 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:39.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:39.125 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:39.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:39.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:39.128 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:39.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155619 -lt 1725155671 ]] 2024-09-01T01:53:39.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:39.129 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:39.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:39.325 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:39.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:39.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:39.328 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:39.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155619 -lt 1725155671 ]] 2024-09-01T01:53:39.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:39.329 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:39.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:39.524 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:39.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:39.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:39.527 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:39.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155619 -lt 1725155671 ]] 2024-09-01T01:53:39.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:39.528 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:39.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:39.724 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.724 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:39.724 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:39.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2024-09-01T01:53:39.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-01T01:53:39.727 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2024-09-01T01:53:39.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1725155619 -lt 1725155671 ]] 2024-09-01T01:53:39.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-01T01:53:39.728 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-01T01:53:54.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2024-09-01T01:53:54.934 INFO:tasks.workunit.client.0.smithi037.stderr:{"election_epoch":12,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:54.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa51ec55-776e-43bf-ab48-832ca942c410","modified":"2024-09-01T01:52:43.393903Z","created":"2024-09-01T01:52:43.393903Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-01T01:53:54.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2024-09-01T01:53:54.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-01T01:53:55.234 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d) after stopping a 2024-09-01T01:53:55.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-01T01:53:55.457 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2024-09-01T01:53:55.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-01T01:53:55.669 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2024-09-01T01:53:55.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-01T01:53:58.824 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2024-09-01T01:53:58.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-01T01:54:02.024 INFO:tasks.workunit.client.0.smithi037.stderr:quorum should be preserved (b,c,d) after stopping e 2024-09-01T01:54:02.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-01T01:54:02.331 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-01T01:54:02.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-01T01:54:02.631 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.a 2024-09-01T01:54:02.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-01T01:54:02.848 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-01T01:54:02.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-01T01:54:06.002 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-01T01:54:06.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-01T01:54:06.267 INFO:tasks.workunit.client.0.smithi037.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-01T01:54:06.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-01T01:54:09.463 INFO:tasks.workunit.client.0.smithi037.stderr:safe to remove mon.e 2024-09-01T01:54:09.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-01T01:54:09.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:54:09.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:54:09.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:54:09.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:54:09.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:54:09.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:54:09.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:54:09.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:54:09.587 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:54:09.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:54:09.588 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:54:09.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:54:09.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:54:09.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:54:09.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:54:09.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:54:09.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:54:09.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:54:09.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:54:09.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:54:09.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:09.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:54:09.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ok-to-stop 0 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2024-09-01T01:54:09.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:54:09.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:54:09.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:54:09.606 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:54:09.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:54:09.607 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:54:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:54:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:54:09.608 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:54:09.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:54:09.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:54:09.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:54:09.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T01:54:09.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2024-09-01T01:54:09.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:54:09.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:09.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.42777 2024-09-01T01:54:09.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.42777 2024-09-01T01:54:09.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:54:09.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:54:09.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T01:54:09.614 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T01:54:09.615 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T01:54:09.679 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-01T01:54:09.680 DEBUG:teuthology.orchestra.run.smithi037:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-01T01:54:09.741 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T01:54:09.747 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T01:54:09.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:54:09.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/rados-striper 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-01T01:54:09.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none ' 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:54:09.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:54:09.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:54:09.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:54:09.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:54:09.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:54:09.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:54:09.752 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:54:09.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:54:09.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:54:09.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-01T01:54:09.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:54:09.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:09.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:09.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.59843 2024-09-01T01:54:09.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:54:09.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:54:09.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2024-09-01T01:54:09.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:54:09.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:09.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:09.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.59843 2024-09-01T01:54:09.758 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:54:09.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T01:54:09.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T01:54:09.758 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2024-09-01T01:54:09.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-01T01:54:10.168 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:54:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:54:10.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:54:10.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.59843/ceph-mon.a.asok 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:54:10.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.59843/ceph-mon.a.asok config get fsid 2024-09-01T01:54:10.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:54:10.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.59843/ceph-mon.a.asok 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:54:10.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.59843/ceph-mon.a.asok config get mon_host 2024-09-01T01:54:10.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-01T01:54:10.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-01T01:54:10.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:54:10.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:54:10.529 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:54:10.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:54:10.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2024-09-01T01:54:10.532 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:54:10.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b74f05f0-8f3d-4013-87a1-9ea060e32865 2024-09-01T01:54:10.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b74f05f0-8f3d-4013-87a1-9ea060e32865' 2024-09-01T01:54:10.533 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 b74f05f0-8f3d-4013-87a1-9ea060e32865 2024-09-01T01:54:10.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:54:10.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCydNmiAR8IBAA4qmdD8xTaiNmguXqvOiFZQ== 2024-09-01T01:54:10.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCydNmiAR8IBAA4qmdD8xTaiNmguXqvOiFZQ=="}' 2024-09-01T01:54:10.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b74f05f0-8f3d-4013-87a1-9ea060e32865 -i td/rados-striper/0/new.json 2024-09-01T01:54:11.402 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:54:11.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2024-09-01T01:54:11.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCydNmiAR8IBAA4qmdD8xTaiNmguXqvOiFZQ== --osd-uuid b74f05f0-8f3d-4013-87a1-9ea060e32865 2024-09-01T01:54:11.441 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:11.439+0000 7ff7fbc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:11.470 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:11.469+0000 7ff7fbc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:11.471 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:11.470+0000 7ff7fbc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:11.472 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:11.470+0000 7ff7fbc14640 -1 bdev(0x55c950243c00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:54:11.472 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:11.470+0000 7ff7fbc14640 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-01T01:54:17.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-01T01:54:17.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:54:17.175 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:54:17.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:54:17.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:54:17.386 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:54:17.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:54:17.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:54:17.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:54:17.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:54:17.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:54:17.411 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:17.409+0000 7fa9e4dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:17.413 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:17.412+0000 7fa9e4dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:17.414 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:17.413+0000 7fa9e4dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:17.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:54:17.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:54:17.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:54:17.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:54:17.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:54:17.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:17.593 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:54:17.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:54:17.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:17.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:17.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:18.504 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:18.503+0000 7fa9e4dc5640 -1 Falling back to public interface 2024-09-01T01:54:18.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:18.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:18.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:54:18.811 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:54:18.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:18.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:19.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:20.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:20.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:20.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:54:20.012 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:54:20.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:20.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:20.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:21.124 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:21.121+0000 7fa9e4dc5640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:54:21.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:21.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:21.213 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:54:21.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:54:21.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:21.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:21.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:22.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:22.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:22.596 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:54:22.596 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:54:22.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:22.597 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:22.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:23.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:23.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:23.799 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:54:23.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:54:23.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:23.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:24.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:25.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:25.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:25.002 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:54:25.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:54:25.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:25.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:25.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:26.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:26.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:26.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:54:26.204 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:54:26.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:26.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:26.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:27.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:27.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:27.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:54:27.410 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:54:27.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:27.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:27.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:28.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:28.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:28.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:54:28.612 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:54:28.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:28.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:28.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:29.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:29.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:29.814 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:54:29.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:54:29.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:29.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:30.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:31.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:31.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:31.016 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:54:31.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:54:31.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:31.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:31.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:32.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:54:32.220 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:54:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:32.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:32.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:33.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:33.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:33.423 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:54:33.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:54:33.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:33.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:33.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:34.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:34.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:34.625 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:54:34.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:54:34.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:34.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:34.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:35.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:35.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:35.827 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:54:35.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:54:35.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:35.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:36.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:37.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:37.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:37.028 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:54:37.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:54:37.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:37.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:37.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:38.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:38.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:38.230 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:54:38.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:54:38.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:38.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:38.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:39.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:39.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:39.431 INFO:tasks.workunit.client.0.smithi037.stdout:18 2024-09-01T01:54:39.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T01:54:39.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:39.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:39.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:40.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:40.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:40.633 INFO:tasks.workunit.client.0.smithi037.stdout:19 2024-09-01T01:54:40.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T01:54:40.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:40.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:40.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:41.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:41.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:41.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-01T01:54:41.837 INFO:tasks.workunit.client.0.smithi037.stdout:20 2024-09-01T01:54:41.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:41.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:54:42.076 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1841254758,v1:127.0.0.1:6801/1841254758] [v2:127.0.0.1:6802/1841254758,v1:127.0.0.1:6803/1841254758] exists,up b74f05f0-8f3d-4013-87a1-9ea060e32865 2024-09-01T01:54:42.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:54:42.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:54:42.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:54:42.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-01T01:54:42.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:54:42.083 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:54:42.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3eb4bf54-fad5-42a2-8f63-13e9d63bb01c 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 3eb4bf54-fad5-42a2-8f63-13e9d63bb01c 2024-09-01T01:54:42.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3eb4bf54-fad5-42a2-8f63-13e9d63bb01c' 2024-09-01T01:54:42.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:54:42.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBiydNmy2y8BRAAgwtxZ490HZ+vBZNclgcfKQ== 2024-09-01T01:54:42.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBiydNmy2y8BRAAgwtxZ490HZ+vBZNclgcfKQ=="}' 2024-09-01T01:54:42.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3eb4bf54-fad5-42a2-8f63-13e9d63bb01c -i td/rados-striper/1/new.json 2024-09-01T01:54:42.577 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:54:42.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2024-09-01T01:54:42.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiydNmy2y8BRAAgwtxZ490HZ+vBZNclgcfKQ== --osd-uuid 3eb4bf54-fad5-42a2-8f63-13e9d63bb01c 2024-09-01T01:54:42.614 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:42.613+0000 7f6d5232e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:42.615 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:42.615+0000 7f6d5232e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:42.616 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:42.616+0000 7f6d5232e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:42.617 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:42.616+0000 7f6d5232e640 -1 bdev(0x561b10de3c00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2024-09-01T01:54:42.617 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:42.617+0000 7f6d5232e640 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-01T01:54:48.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-01T01:54:48.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:54:48.303 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T01:54:48.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T01:54:48.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:54:48.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T01:54:48.514 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T01:54:48.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:54:48.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:54:48.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:54:48.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:54:48.539 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:48.538+0000 7fa45240b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:48.544 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:48.543+0000 7fa45240b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:48.545 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:48.545+0000 7fa45240b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:48.725 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:54:48.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:54:48.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:48.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:48.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:49.654 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:49.654+0000 7fa45240b640 -1 Falling back to public interface 2024-09-01T01:54:49.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:49.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:49.936 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:54:49.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:54:49.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:49.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:50.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:51.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:51.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:51.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:54:51.137 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:54:51.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:51.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:51.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:52.296 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:54:52.295+0000 7fa45240b640 -1 osd.1 0 log_to_monitors true 2024-09-01T01:54:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:52.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:54:52.339 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:54:52.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:52.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:52.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:53.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:53.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:53.798 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:54:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:54:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:53.998 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:54.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:55.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:55.000 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:54:55.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:54:55.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:55.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:55.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:56.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:56.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:56.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:54:56.201 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:54:56.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:56.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:56.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:57.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:57.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:57.404 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:54:57.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:54:57.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:57.405 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:57.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:58.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:58.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:58.607 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:54:58.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:54:58.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:58.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:54:58.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:54:59.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:54:59.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:54:59.808 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:54:59.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:54:59.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:54:59.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:00.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:01.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:01.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:01.010 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:55:01.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:55:01.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:01.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:01.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:02.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:02.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:02.214 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:55:02.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:55:02.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:02.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:02.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:03.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:03.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:03.416 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:55:03.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:55:03.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:03.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:03.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:04.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:04.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:04.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:55:04.618 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:55:04.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:04.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:04.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:05.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:05.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:05.821 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:55:05.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:55:05.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:05.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:06.023 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:07.023 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:07.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:07.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:55:07.024 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:55:07.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:07.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:07.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:08.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:08.226 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:55:08.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:55:08.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:08.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:08.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:09.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:09.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:09.428 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:55:09.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:55:09.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:09.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:09.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:10.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:10.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:10.629 INFO:tasks.workunit.client.0.smithi037.stdout:18 2024-09-01T01:55:10.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T01:55:10.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:10.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:10.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:11.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:11.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:11.829 INFO:tasks.workunit.client.0.smithi037.stdout:19 2024-09-01T01:55:11.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T01:55:11.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:11.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:12.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:13.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:13.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:13.033 INFO:tasks.workunit.client.0.smithi037.stdout:20 2024-09-01T01:55:13.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-01T01:55:13.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:13.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T01:55:13.235 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/147687239,v1:127.0.0.1:6809/147687239] [v2:127.0.0.1:6810/147687239,v1:127.0.0.1:6811/147687239] exists,up 3eb4bf54-fad5-42a2-8f63-13e9d63bb01c 2024-09-01T01:55:13.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:55:13.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:55:13.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:55:13.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-01T01:55:13.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:55:13.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:55:13.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:55:13.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2024-09-01T01:55:13.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:55:13.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=623feee7-196d-4b0b-bacd-577fbf62ca39 2024-09-01T01:55:13.243 INFO:tasks.workunit.client.0.smithi037.stdout:add osd2 623feee7-196d-4b0b-bacd-577fbf62ca39 2024-09-01T01:55:13.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 623feee7-196d-4b0b-bacd-577fbf62ca39' 2024-09-01T01:55:13.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:55:13.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBydNm4TU9DxAANgRV2BQkO7uPktDw/RdjfA== 2024-09-01T01:55:13.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBydNm4TU9DxAANgRV2BQkO7uPktDw/RdjfA=="}' 2024-09-01T01:55:13.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 623feee7-196d-4b0b-bacd-577fbf62ca39 -i td/rados-striper/2/new.json 2024-09-01T01:55:13.713 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:55:13.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2024-09-01T01:55:13.724 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBydNm4TU9DxAANgRV2BQkO7uPktDw/RdjfA== --osd-uuid 623feee7-196d-4b0b-bacd-577fbf62ca39 2024-09-01T01:55:13.751 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:13.750+0000 7fed46e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:13.752 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:13.752+0000 7fed46e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:13.753 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:13.753+0000 7fed46e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:13.754 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:13.753+0000 7fed46e1c640 -1 bdev(0x557b62f3dc00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2024-09-01T01:55:13.754 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:13.754+0000 7fed46e1c640 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-01T01:55:19.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-01T01:55:19.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:55:19.460 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd2 key to auth repository 2024-09-01T01:55:19.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T01:55:19.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:55:19.668 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.2 2024-09-01T01:55:19.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T01:55:19.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=54e9e80b-6969-489c-ad6d-584c4f73c3a7 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.59843/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:55:19.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:55:19.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:55:19.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:55:19.693 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:19.691+0000 7faa3a40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:19.707 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:19.706+0000 7faa3a40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:19.709 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:19.707+0000 7faa3a40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:19.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:55:19.889 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:55:19.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:19.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:20.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:21.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:21.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:21.108 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:55:21.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:55:21.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:21.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:21.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:21.332 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:21.331+0000 7faa3a40d640 -1 Falling back to public interface 2024-09-01T01:55:22.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:22.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:22.310 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:55:22.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:55:22.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:22.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:22.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:23.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:23.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:23.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:55:23.510 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:55:23.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:23.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:23.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:24.510 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:24.508+0000 7faa3a40d640 -1 osd.2 0 log_to_monitors true 2024-09-01T01:55:24.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:24.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:24.712 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:55:24.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:55:24.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:24.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:25.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:26.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:26.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:26.110 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:55:26.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:55:26.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:26.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:26.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:27.313 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:55:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:55:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:27.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:27.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:28.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:28.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:28.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:55:28.518 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:55:28.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:28.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:28.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:29.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:29.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:29.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:55:29.720 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:55:29.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:29.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:29.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:30.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:30.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:30.923 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:55:30.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:55:30.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:30.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:31.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:32.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:32.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:32.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:55:32.128 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:55:32.128 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:32.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:32.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:33.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:33.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:33.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:55:33.330 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:55:33.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:33.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:33.530 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:34.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:34.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:34.532 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:55:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:55:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:34.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:35.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:35.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:35.736 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:55:35.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:55:35.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:35.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:35.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:36.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:36.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:36.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:55:36.938 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:55:36.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:36.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:37.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:38.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:38.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:38.142 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:55:38.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:55:38.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:38.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:38.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:39.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:39.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:39.343 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:55:39.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:55:39.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:39.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:39.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:40.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:40.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:40.546 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:55:40.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:55:40.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:40.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:40.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:41.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:41.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:41.751 INFO:tasks.workunit.client.0.smithi037.stdout:18 2024-09-01T01:55:41.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T01:55:41.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:41.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:41.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:42.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:42.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:42.952 INFO:tasks.workunit.client.0.smithi037.stdout:19 2024-09-01T01:55:42.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T01:55:42.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:42.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:43.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:44.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:44.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:44.154 INFO:tasks.workunit.client.0.smithi037.stdout:20 2024-09-01T01:55:44.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-01T01:55:44.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:44.154 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:44.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:45.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:45.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:45.357 INFO:tasks.workunit.client.0.smithi037.stdout:21 2024-09-01T01:55:45.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-01T01:55:45.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:45.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:45.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:55:46.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:55:46.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:55:46.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-01T01:55:46.579 INFO:tasks.workunit.client.0.smithi037.stdout:22 2024-09-01T01:55:46.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:55:46.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T01:55:46.797 INFO:tasks.workunit.client.0.smithi037.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1038282794,v1:127.0.0.1:6817/1038282794] [v2:127.0.0.1:6818/1038282794,v1:127.0.0.1:6819/1038282794] exists,up 623feee7-196d-4b0b-bacd-577fbf62ca39 2024-09-01T01:55:46.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:55:46.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:55:46.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:55:46.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-01T01:55:46.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T01:55:47.023 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T01:55:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T01:55:47.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T01:55:47.413 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T01:55:47.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T01:55:48.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T01:55:50.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-01T01:55:50.420 INFO:tasks.workunit.client.0.smithi037.stderr:1+0 records in 2024-09-01T01:55:50.420 INFO:tasks.workunit.client.0.smithi037.stderr:1+0 records out 2024-09-01T01:55:50.420 INFO:tasks.workunit.client.0.smithi037.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 6.8995e-05 s, 17.9 MB/s 2024-09-01T01:55:50.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-01T01:55:51.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-01T01:55:51.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-01T01:55:51.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-01T01:55:51.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-01T01:55:51.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-01T01:55:51.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-01T01:55:51.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-01T01:55:51.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-01T01:55:51.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-01T01:55:51.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-01T01:55:51.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-01T01:55:51.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-01T01:55:51.537 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-01T01:55:51.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-01T01:55:52.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-01T01:55:52.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-01T01:55:52.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-01T01:55:52.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-01T01:55:52.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-01T01:55:52.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-01T01:55:52.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-01T01:55:52.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-01T01:55:52.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-01T01:55:52.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-01T01:55:52.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-01T01:55:52.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-01T01:55:52.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-01T01:55:52.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-01T01:55:52.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-01T01:55:52.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-01T01:55:52.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-01T01:55:52.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-01T01:55:52.343 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-01T01:55:52.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-01T01:55:52.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-01T01:55:52.461 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No data available' 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:55:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-01T01:55:52.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:55:52.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:55:52.492 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-01T01:55:52.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T01:55:52.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-01T01:55:52.493 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-01T01:55:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:55:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No data available' 2024-09-01T01:55:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:55:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:55:52.494 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-01T01:55:52.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:55:52.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:55:52.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-01T01:55:52.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T01:55:52.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No such file or directory' 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:55:52.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:55:52.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-01T01:55:52.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:55:52.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:55:52.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/rados-striper 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local 'expected=No such file or directory' 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:55:52.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-01T01:55:52.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:55:52.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:55:52.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:55:52.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:55:52.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:55:52.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:55:52.928 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:55:52.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:55:52.929 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:55:52.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:55:52.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:55:52.930 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:55:52.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:52.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:55:52.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:55:52.931 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:52.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:55:52.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:55:52.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-01T01:55:52.944 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:55:52.944 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:52.944 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:55:52.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.59843 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/rados-striper 0 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:55:52.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:55:52.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:55:52.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:55:52.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:55:52.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:55:52.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:55:52.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:55:52.948 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:55:52.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:55:52.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:55:52.950 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:55:52.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:52.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:55:52.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:55:52.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:52.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:55:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T01:55:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2024-09-01T01:55:52.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:55:52.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:52.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.59843 2024-09-01T01:55:52.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.59843 2024-09-01T01:55:52.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:55:52.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:55:52.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T01:55:52.956 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T01:55:52.956 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T01:55:53.020 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-01T01:55:53.020 DEBUG:teuthology.orchestra.run.smithi037:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-01T01:55:53.085 INFO:tasks.workunit.client.0.smithi037.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T01:55:53.089 INFO:tasks.workunit.client.0.smithi037.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T01:55:53.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T01:55:53.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2336: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-09-01T01:55:53.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-01T01:55:53.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-01T01:55:53.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: export CEPH_ARGS 2024-09-01T01:55:53.090 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: uuidgen 2024-09-01T01:55:53.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: CEPH_ARGS+=' --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none ' 2024-09-01T01:55:53.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2341: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-01T01:55:53.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2342: run_tests: export CEPH_CONF=/dev/null 2024-09-01T01:55:53.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2342: run_tests: CEPH_CONF=/dev/null 2024-09-01T01:55:53.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: set 2024-09-01T01:55:53.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: local 'funcs=test_activate_osd 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:test_activate_osd_after_mark_down 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:test_activate_osd_skip_benchmark 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:test_destroy_osd 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:test_display_logs 2024-09-01T01:55:53.093 INFO:tasks.workunit.client.0.smithi037.stderr:test_erasure_code_plugin_exists 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_expect_failure 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_flush_pg_stats 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_config 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_is_making_recovery_progress 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_last_scrub_stamp 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_not_primary 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_num_active_clean 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_num_active_or_peered 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_num_pgs 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_op_scheduler 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_osd_id_used_by_pgs 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_osds 2024-09-01T01:55:53.094 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_pg 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_primary 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_get_timeout_delays 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_is_clean 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_kill_daemon 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_kill_daemons 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_objectstore_tool 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_pg_schedule_scrub 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_pg_scrub 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_repair 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_run_in_background 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_run_mon 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_run_osd 2024-09-01T01:55:53.095 INFO:tasks.workunit.client.0.smithi037.stderr:test_set_config 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_setup 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_teardown 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_background 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_for_clean 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_for_health_ok 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_for_osd 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_for_peered 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_for_scrub 2024-09-01T01:55:53.096 INFO:tasks.workunit.client.0.smithi037.stderr:test_wait_osd_id_used_by_pgs' 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: run_tests: local dir=td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:55:53.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:55:53.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:55:53.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:55:53.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:55:53.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:55:53.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:55:53.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:55:53.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:55:53.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:55:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:55:53.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:55:53.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:53.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:53.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:55:53.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:55:53.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:53.594 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:55:53.595 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:55:53.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:55:53.796 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:55:53.796 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:55:53.796 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:55:53.796 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:55:53.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:55:53.797 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:55:53.797 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:55:53.797 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:55:53.798 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:55:53.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:55:53.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:55:53.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:55:53.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:55:53.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:55:53.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-01T01:55:53.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:55:53.962 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:55:53.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:55:53.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:55:53.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:55:53.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:54.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:55:54.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:55:54.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:55:54.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-01T01:55:54.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:55:54.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:55:54.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:55:54.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:55:54.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:55:54.854 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:55:54.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=294de878-f25a-43cc-818a-8e33ff00ae3c 2024-09-01T01:55:54.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 294de878-f25a-43cc-818a-8e33ff00ae3c' 2024-09-01T01:55:54.855 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 294de878-f25a-43cc-818a-8e33ff00ae3c 2024-09-01T01:55:54.855 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:55:54.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCqydNm2jDkMxAA72M1Z8qBF64kjjTZQqk3Og== 2024-09-01T01:55:54.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCqydNm2jDkMxAA72M1Z8qBF64kjjTZQqk3Og=="}' 2024-09-01T01:55:54.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 294de878-f25a-43cc-818a-8e33ff00ae3c -i td/ceph-helpers/0/new.json 2024-09-01T01:55:55.203 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:55:55.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:55:55.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqydNm2jDkMxAA72M1Z8qBF64kjjTZQqk3Og== --osd-uuid 294de878-f25a-43cc-818a-8e33ff00ae3c 2024-09-01T01:55:55.246 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:55.246+0000 7f4f1d40c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:55.288 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:55.287+0000 7f4f1d40c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:55.289 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:55.289+0000 7f4f1d40c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:55:55.290 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:55.289+0000 7f4f1d40c640 -1 bdev(0x55fc295e7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:55:55.290 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:55:55.289+0000 7f4f1d40c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:56:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:56:00.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:56:00.735 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:56:00.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:56:00.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:56:01.324 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:56:01.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:56:01.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:56:01.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:56:01.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:56:01.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:56:01.351 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:01.351+0000 7f1829e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:01.377 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:01.377+0000 7f1829e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:01.378 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:01.378+0000 7f1829e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:01.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:56:01.732 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:56:01.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:01.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:02.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:02.711 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:02.711+0000 7f1829e0e640 -1 Falling back to public interface 2024-09-01T01:56:03.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:03.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:03.158 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:56:03.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:56:03.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:03.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:03.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:04.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:04.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:04.601 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:56:04.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:56:04.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:04.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:05.050 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:05.656 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:05.654+0000 7f1829e0e640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:56:06.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:06.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:06.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:56:06.052 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:56:06.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:06.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:06.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:07.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:07.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:07.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:56:07.675 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:56:07.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:07.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:08.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:09.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:09.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:09.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:56:09.095 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:56:09.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:09.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:09.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:10.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:10.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:10.554 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:56:10.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:56:10.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:10.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:10.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:11.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:11.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:11.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:56:11.993 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:56:11.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:11.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:12.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:13.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:13.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:13.426 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:56:13.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:56:13.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:13.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:13.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:14.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:14.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:14.855 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:56:14.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:56:14.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:14.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:15.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:16.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:16.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:16.381 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:56:16.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:56:16.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:16.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:16.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:17.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:17.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:17.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:56:17.797 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:56:17.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:17.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:18.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:19.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:19.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:19.396 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:56:19.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:56:19.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:19.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:20.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:21.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:21.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:21.351 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:56:21.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:56:21.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:21.351 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:22.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:23.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:23.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:23.136 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:56:23.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:56:23.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:23.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:23.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:24.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:24.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:24.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:56:24.620 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:56:24.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:24.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:25.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:26.539 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:26.539 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:26.539 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:56:26.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:56:26.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:26.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:27.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:28.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:28.418 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:56:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:56:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:28.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:28.833 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3299591250,v1:127.0.0.1:6803/3299591250] [v2:127.0.0.1:6804/3299591250,v1:127.0.0.1:6805/3299591250] exists,up 294de878-f25a-43cc-818a-8e33ff00ae3c 2024-09-01T01:56:28.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:56:28.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2024-09-01T01:56:28.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:56:28.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-01T01:56:28.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-01T01:56:28.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2024-09-01T01:56:28.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-01T01:56:28.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:56:28.922 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:56:28.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:56:28.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:56:28.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:56:29.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:56:29.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-01T01:56:29.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-01T01:56:29.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T01:56:29.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T01:56:29.231 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:56:29.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:56:29.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T01:56:29.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-01T01:56:29.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:56:29.233 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T01:56:29.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T01:56:29.257 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:29.256+0000 7f0ffd80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:29.279 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:29.278+0000 7f0ffd80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:29.280 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:29.279+0000 7f0ffd80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:29.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T01:56:29.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:56:29.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:56:29.667 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:29.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:30.144 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:30.142+0000 7f0ffd80c640 -1 Falling back to public interface 2024-09-01T01:56:30.144 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:31.145 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:31.145 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:31.145 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:56:31.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:56:31.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:31.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:31.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:32.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:32.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:32.588 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:56:32.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:56:32.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:32.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:32.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:33.599 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:33.598+0000 7f0ffd80c640 -1 osd.0 6 log_to_monitors true 2024-09-01T01:56:33.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:33.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:33.997 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:56:33.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:56:33.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:33.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:34.444 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2763648849,v1:127.0.0.1:6803/2763648849] [v2:127.0.0.1:6804/2763648849,v1:127.0.0.1:6805/2763648849] exists,up 294de878-f25a-43cc-818a-8e33ff00ae3c 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2024-09-01T01:56:34.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:56:34.533 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:56:34.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:56:34.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:56:34.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:56:34.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:56:34.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:56:34.638 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:56:34.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:56:34.639 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:56:34.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:56:34.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:56:34.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:56:34.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:56:34.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:56:34.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:56:34.641 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:56:34.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:56:34.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:56:34.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:56:34.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:56:34.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:34.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:34.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:56:34.651 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:56:34.652 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:56:34.652 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:56:34.652 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:56:34.652 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:56:34.652 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:56:34.653 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:56:34.653 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:56:34.653 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:56:34.653 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:56:34.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:56:34.654 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:56:34.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:56:34.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:56:34.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:56:34.655 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:56:34.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:56:34.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:56:34.656 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:56:34.658 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:56:34.660 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:56:34.660 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:34.660 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:34.660 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:56:34.661 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:56:34.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:56:35.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:35.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:56:35.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:56:35.300 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:56:35.301 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:56:35.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:56:35.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:56:35.390 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:56:35.391 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:56:35.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:56:35.470 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:36.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:56:36.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:56:36.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:56:36.358 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:56:36.359 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:56:36.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:56:36.360 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:56:36.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9a368019-d5be-4732-9850-108a108acd04' 2024-09-01T01:56:36.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:56:36.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUydNmS2ZvFhAABj2VIWqKplhmkkjVA3y04Q== 2024-09-01T01:56:36.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUydNmS2ZvFhAABj2VIWqKplhmkkjVA3y04Q=="}' 2024-09-01T01:56:36.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9a368019-d5be-4732-9850-108a108acd04 -i td/ceph-helpers/0/new.json 2024-09-01T01:56:36.649 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:56:36.659 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:56:36.660 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUydNmS2ZvFhAABj2VIWqKplhmkkjVA3y04Q== --osd-uuid 9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:56:36.689 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:36.688+0000 7f39b5619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:36.718 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:36.717+0000 7f39b5619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:36.719 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:36.718+0000 7f39b5619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:36.719 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:36.719+0000 7f39b5619640 -1 bdev(0x555cc9c9bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:56:36.719 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:36.719+0000 7f39b5619640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:56:42.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:56:42.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:56:42.384 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:56:42.384 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:56:42.384 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:56:42.952 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:56:42.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:56:42.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:56:42.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:56:42.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:56:42.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:56:42.979 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:42.978+0000 7fcbf6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:42.997 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:42.997+0000 7fcbf6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:42.998 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:42.998+0000 7fcbf6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:56:43.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:56:43.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:56:43.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:56:43.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:56:43.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:56:43.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:43.420 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:56:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:56:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:43.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:44.105 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:44.105+0000 7fcbf6d58640 -1 Falling back to public interface 2024-09-01T01:56:44.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:44.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:44.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:56:44.855 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:56:44.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:44.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:45.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:46.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:46.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:46.306 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:56:46.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:56:46.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:46.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:46.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:47.342 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:56:47.342+0000 7fcbf6d58640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:56:47.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:47.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:47.757 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:56:47.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:56:47.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:47.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:48.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:49.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:49.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:49.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:56:49.416 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:56:49.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:49.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:49.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:56:50.836 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:56:50.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:50.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:51.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:52.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:52.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:52.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:56:52.258 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:56:52.258 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:52.258 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:52.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:53.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:53.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:53.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:56:53.689 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:56:53.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:53.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:54.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:55.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:55.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:55.204 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:56:55.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:56:55.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:55.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:55.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:56.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:56.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:56.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:56:56.649 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:56:56.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:56.650 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:57.092 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:58.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:58.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:58.094 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:56:58.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:56:58.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:58.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:58.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:56:59.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:56:59.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:56:59.512 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:56:59.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:56:59.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:56:59.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:56:59.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:00.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:00.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:00.950 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:57:00.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:57:00.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:00.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:01.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:02.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:02.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:02.828 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:57:02.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:57:02.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:02.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:03.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:04.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:04.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:04.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:57:04.842 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:57:04.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:04.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:05.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:06.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:06.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:06.273 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:57:06.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:57:06.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:06.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:06.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:07.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:07.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:07.770 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:57:07.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:57:07.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:07.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:08.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:09.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:09.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:09.542 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:57:09.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:57:09.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:09.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:10.028 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3013998031,v1:127.0.0.1:6803/3013998031] [v2:127.0.0.1:6804/3013998031,v1:127.0.0.1:6805/3013998031] exists,up 9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:57:10.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:10.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:10.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-01T01:57:10.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:57:10.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-01T01:57:10.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-01T01:57:10.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-01T01:57:10.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-01T01:57:10.117 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:10.117 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:10.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:10.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:10.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:10.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:10.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-01T01:57:10.861 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already down. 2024-09-01T01:57:10.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-01T01:57:10.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-01T01:57:10.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:10.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:10.872 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:10.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:10.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:10.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:10.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:10.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-01T01:57:11.306 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3013998031,v1:127.0.0.1:6803/3013998031] [v2:127.0.0.1:6804/3013998031,v1:127.0.0.1:6805/3013998031] exists 9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:57:11.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:11.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:11.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:11.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-01T01:57:11.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:57:11.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:57:11.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:57:11.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:57:11.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T01:57:11.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-01T01:57:11.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:57:11.313 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T01:57:11.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T01:57:11.336 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:11.336+0000 7f57c9885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:11.338 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:11.338+0000 7f57c9885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:11.339 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:11.339+0000 7f57c9885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:11.759 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:11.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:11.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:11.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:12.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:12.714 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:12.714+0000 7f57c9885640 -1 Falling back to public interface 2024-09-01T01:57:13.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:13.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:13.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:57:13.211 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:57:13.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:13.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:13.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:14.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:14.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:14.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:57:14.672 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:57:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:15.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:15.983 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:15.982+0000 7f57c9885640 -1 osd.0 5 log_to_monitors true 2024-09-01T01:57:16.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:16.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:16.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:57:16.078 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:57:16.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:16.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:16.557 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1975929632,v1:127.0.0.1:6803/1975929632] [v2:127.0.0.1:6804/1975929632,v1:127.0.0.1:6805/1975929632] exists,up 9a368019-d5be-4732-9850-108a108acd04 2024-09-01T01:57:16.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:16.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:16.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:16.558 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-01T01:57:16.559 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_max_backfills 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:57:16.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:16.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:16.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:16.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:16.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:16.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:16.751 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:57:16.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:57:16.752 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:57:16.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:57:16.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:57:16.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:57:16.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:16.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:57:16.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:57:16.754 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:16.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:57:16.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:57:16.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:57:16.761 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:57:16.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:16.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:16.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:57:16.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:57:16.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:16.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:16.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:16.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:16.765 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:57:16.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:57:16.766 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:57:16.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:57:16.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:57:16.767 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:57:16.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:16.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:57:16.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:57:16.768 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:16.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:57:16.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:57:16.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:57:16.771 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:57:16.771 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:16.771 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:16.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:57:16.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:57:16.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:57:16.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:57:16.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:57:16.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:16.773 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:16.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-01T01:57:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:57:16.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:57:16.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:57:16.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:57:16.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:57:16.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:57:17.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:57:17.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:17.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:17.249 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:17.250 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:17.250 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:17.250 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:17.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:57:17.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:57:17.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:57:17.442 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:57:17.443 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:57:17.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:17.530 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:17.531 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:57:17.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:57:17.531 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:57:17.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:57:17.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:57:18.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:57:18.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:57:18.475 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:57:18.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:57:18.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:18.510 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-01T01:57:18.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:57:18.512 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:57:18.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:18.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 67740c06-1d16-4239-8935-877431637b64' 2024-09-01T01:57:18.512 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:18.512 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:57:18.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+ydNmyR5GHxAAEaFePvvAPmpn0IkpeF3+dA== 2024-09-01T01:57:18.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+ydNmyR5GHxAAEaFePvvAPmpn0IkpeF3+dA=="}' 2024-09-01T01:57:18.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 67740c06-1d16-4239-8935-877431637b64 -i td/ceph-helpers/0/new.json 2024-09-01T01:57:18.824 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:18.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:57:18.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQD+ydNmyR5GHxAAEaFePvvAPmpn0IkpeF3+dA== --osd-uuid 67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:18.866 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:18.865+0000 7f4c34b02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:18.893 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:18.892+0000 7f4c34b02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:18.894 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:18.894+0000 7f4c34b02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:18.895 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:18.894+0000 7f4c34b02640 -1 bdev(0x565086e4bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:57:18.895 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:18.894+0000 7f4c34b02640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:57:24.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:57:24.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:57:24.606 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:57:24.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:57:24.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:57:25.185 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:57:25.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:57:25.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-01T01:57:25.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:57:25.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:57:25.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:57:25.214 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:25.213+0000 7f73d30ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:25.216 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:25.215+0000 7f73d30ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:25.217 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:25.216+0000 7f73d30ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:25.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:25.629 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:25.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:25.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:26.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:26.608 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:26.608+0000 7f73d30ef640 -1 Falling back to public interface 2024-09-01T01:57:27.072 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:27.072 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:27.072 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:57:27.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:57:27.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:27.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:27.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:28.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:28.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:28.523 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:57:28.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:57:28.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:28.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:28.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:29.451 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:29.450+0000 7f73d30ef640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:57:29.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:29.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:29.936 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:57:29.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:57:29.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:29.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:30.455 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/343138854,v1:127.0.0.1:6803/343138854] [v2:127.0.0.1:6804/343138854,v1:127.0.0.1:6805/343138854] exists,up 67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:30.457 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:30.458 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:30.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-01T01:57:30.458 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-01T01:57:30.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-01T01:57:30.544 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-01T01:57:30.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-01T01:57:30.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-01T01:57:30.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:30.632 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:30.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:30.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:30.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:30.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:30.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-01T01:57:31.383 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already down. 2024-09-01T01:57:31.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-01T01:57:31.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2024-09-01T01:57:31.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:31.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:31.395 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:31.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:31.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:31.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:31.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:31.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2024-09-01T01:57:31.836 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/343138854,v1:127.0.0.1:6803/343138854] [v2:127.0.0.1:6804/343138854,v1:127.0.0.1:6805/343138854] exists 67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:57:31.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:31.841 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-01T01:57:31.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:57:31.843 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:57:31.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T01:57:31.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T01:57:31.867 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:31.866+0000 7fc21763e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:31.869 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:31.868+0000 7fc21763e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:31.870 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:31.869+0000 7fc21763e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:32.271 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:32.272 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:32.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:32.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:32.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:32.969 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:32.969+0000 7fc21763e640 -1 Falling back to public interface 2024-09-01T01:57:33.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:33.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:33.707 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:57:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:57:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:34.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:35.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:35.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:35.159 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:57:35.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:57:35.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:35.159 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:35.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:36.070 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:36.069+0000 7fc21763e640 -1 osd.0 6 log_to_monitors true 2024-09-01T01:57:36.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:36.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:36.575 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:57:36.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:57:36.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:36.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:37.015 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2471168002,v1:127.0.0.1:6803/2471168002] [v2:127.0.0.1:6804/2471168002,v1:127.0.0.1:6805/2471168002] exists,up 67740c06-1d16-4239-8935-877431637b64 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-01T01:57:37.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-01T01:57:37.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-01T01:57:37.104 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-01T01:57:37.104 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:57:37.104 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:57:37.105 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:37.105 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.105 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.105 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:57:37.105 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-01T01:57:37.106 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-01T01:57:37.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:37.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:37.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:37.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:57:37.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:57:37.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:57:37.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:57:37.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:57:37.299 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:57:37.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:37.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:57:37.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:57:37.301 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:37.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:57:37.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:57:37.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:57:37.305 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:57:37.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:57:37.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:57:37.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:57:37.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:57:37.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_destroy_osd td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:57:37.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:57:37.311 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:57:37.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:57:37.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:57:37.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:57:37.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:57:37.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:57:37.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:57:37.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:57:37.317 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:57:37.317 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.317 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:57:37.319 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:57:37.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:57:37.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:57:37.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:57:37.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:57:37.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:57:37.778 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:37.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:57:37.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:57:37.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:57:37.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:57:37.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:57:37.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:57:37.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:57:37.981 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:57:37.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:57:38.063 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:57:38.063 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:57:38.064 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:57:38.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:57:38.144 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:57:39.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:57:39.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:39.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:57:39.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:57:39.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:57:39.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:57:39.039 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:57:39.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=47f6e938-5a67-4692-8bae-bbd3eb8b716b 2024-09-01T01:57:39.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 47f6e938-5a67-4692-8bae-bbd3eb8b716b' 2024-09-01T01:57:39.040 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 47f6e938-5a67-4692-8bae-bbd3eb8b716b 2024-09-01T01:57:39.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:57:39.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATytNmVAxcAxAAyo+MebdwrrVy1dp3B7gp/w== 2024-09-01T01:57:39.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATytNmVAxcAxAAyo+MebdwrrVy1dp3B7gp/w=="}' 2024-09-01T01:57:39.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 47f6e938-5a67-4692-8bae-bbd3eb8b716b -i td/ceph-helpers/0/new.json 2024-09-01T01:57:39.329 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:39.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:57:39.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATytNmVAxcAxAAyo+MebdwrrVy1dp3B7gp/w== --osd-uuid 47f6e938-5a67-4692-8bae-bbd3eb8b716b 2024-09-01T01:57:39.369 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:39.367+0000 7f743731d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:39.397 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:39.396+0000 7f743731d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:39.398 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:39.397+0000 7f743731d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:39.399 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:39.397+0000 7f743731d640 -1 bdev(0x555b9bd9fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:57:39.399 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:39.397+0000 7f743731d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:57:44.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:57:44.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:57:44.945 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:57:44.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:57:44.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:57:45.532 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:57:45.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:57:45.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:57:45.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:57:45.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:57:45.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:57:45.559 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:45.558+0000 7f58e741e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:45.561 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:45.560+0000 7f58e741e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:45.562 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:45.561+0000 7f58e741e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:57:45.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:57:45.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:57:45.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:57:45.966 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:57:45.966 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:57:45.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:57:45.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:45.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:57:45.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:45.967 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:46.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:46.921 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:46.921+0000 7f58e741e640 -1 Falling back to public interface 2024-09-01T01:57:47.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:47.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:47.393 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:57:47.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:57:47.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:47.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:47.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:48.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:48.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:48.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:57:48.845 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:57:48.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:48.845 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:49.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:49.872 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:57:49.872+0000 7f58e741e640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:57:50.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:50.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:50.287 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:57:50.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:57:50.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:50.288 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:50.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:51.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:51.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:51.826 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:57:51.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:57:51.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:51.827 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:52.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:53.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:53.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:53.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:57:53.246 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:57:53.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:53.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:53.753 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:54.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:54.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:54.755 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:57:54.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:57:54.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:54.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:55.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:56.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:56.190 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:57:56.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:56.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:57:56.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:56.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:56.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:57.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:57.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:57.733 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:57:57.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:57:57.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:57.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:58.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:57:59.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:57:59.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:57:59.165 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:57:59.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:57:59.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:57:59.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:57:59.620 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:00.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:00.621 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:00.621 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:58:00.622 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:58:00.622 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:00.622 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:01.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:02.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:02.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:02.059 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:58:02.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:58:02.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:02.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:02.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:03.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:03.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:03.628 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:58:03.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:58:03.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:03.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:04.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:05.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:05.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:05.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:58:05.425 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:58:05.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:05.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:06.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:07.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:07.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:07.563 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:58:07.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:58:07.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:07.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:08.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:09.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:09.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:09.019 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:58:09.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:58:09.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:09.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:09.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:10.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:10.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:10.442 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:58:10.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:58:10.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:10.442 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:11.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:12.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:12.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:12.241 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:58:12.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:58:12.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:12.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:12.746 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2535797429,v1:127.0.0.1:6803/2535797429] [v2:127.0.0.1:6804/2535797429,v1:127.0.0.1:6805/2535797429] exists,up 47f6e938-5a67-4692-8bae-bbd3eb8b716b 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2024-09-01T01:58:12.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2024-09-01T01:58:12.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2024-09-01T01:58:13.372 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already out. 2024-09-01T01:58:13.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-01T01:58:13.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:13.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:13.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:13.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:13.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:13.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:13.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2024-09-01T01:58:14.108 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 is already down. 2024-09-01T01:58:14.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-01T01:58:14.618 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 does not exist 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:14.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:14.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:14.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:14.632 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:14.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:14.633 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:14.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:14.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:14.634 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:14.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:14.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:14.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:14.635 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:14.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:14.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:14.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2024-09-01T01:58:14.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:14.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:14.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:14.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:14.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:14.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:14.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-01T01:58:14.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2024-09-01T01:58:14.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2024-09-01T01:58:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2024-09-01T01:58:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:15.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:15.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:15.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:15.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:15.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:15.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:15.229 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:15.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:15.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:15.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:15.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:15.231 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:15.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:15.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:15.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:15.233 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:15.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:15.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:15.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:15.237 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:15.237 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:15.237 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:15.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:15.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:15.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:15.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_display_logs td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: test_display_logs: local dir=td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: test_display_logs: setup td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:15.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:15.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:15.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:15.242 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:15.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:15.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:15.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:15.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:15.243 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:15.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:15.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:15.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:15.244 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:15.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:15.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:15.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:15.246 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:15.246 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:15.246 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:15.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:15.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:15.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:15.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:58:15.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:58:15.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:15.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:15.249 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:58:15.250 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:58:15.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:58:15.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:58:15.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_display_logs: run_mon td/ceph-helpers a 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:58:15.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:58:15.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:15.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:58:15.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:15.914 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:15.915 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:15.915 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:15.915 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:15.915 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:15.916 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:15.916 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:16.000 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:16.001 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:16.001 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:16.001 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:16.001 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:16.001 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:16.002 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_display_logs: kill_daemons td/ceph-helpers 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:16.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_display_logs: display_logs td/ceph-helpers 2024-09-01T01:58:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: display_logs: local dir=td/ceph-helpers 2024-09-01T01:58:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-01T01:58:17.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: display_logs: read file 2024-09-01T01:58:17.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-01T01:58:17.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-01T01:58:17.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2103: display_logs: read file 2024-09-01T01:58:17.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-01T01:58:17.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_display_logs: teardown td/ceph-helpers 2024-09-01T01:58:17.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:17.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:17.397 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:17.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:17.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:17.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:17.398 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:17.398 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:17.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:17.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:17.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:17.400 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:17.400 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:17.400 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:17.401 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:17.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:17.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:17.404 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:17.404 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:17.404 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:17.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:17.407 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:17.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:17.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:17.409 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:17.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:17.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:17.411 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:17.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:17.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:17.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:17.412 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:17.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:17.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:17.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:17.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:17.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:17.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:17.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:17.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:17.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:17.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:58:17.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:58:17.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:17.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:17.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:58:17.420 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:58:17.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:58:17.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:58:17.893 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:58:17.893 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:17.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:58:18.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:58:18.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:58:18.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:18.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:18.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:18.080 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:18.081 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:18.081 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:58:18.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:58:18.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:18.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:18.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:18.171 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:18.172 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:18.172 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:18.172 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:18.172 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:58:18.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:19.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:19.111 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:58:19.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:58:19.139 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=jerasure 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-01T01:58:19.140 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-01T01:58:19.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s= 2024-09-01T01:58:19.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=0 2024-09-01T01:58:19.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-01T01:58:19.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-01T01:58:19.777 INFO:tasks.workunit.client.0.smithi037.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-01T01:58:19.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 0 2024-09-01T01:58:19.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=FAKE 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-01T01:58:19.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-01T01:58:19.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-01T01:58:19.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-01T01:58:19.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=5 2024-09-01T01:58:19.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-01T01:58:19.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-01T01:58:19.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-01T01:58:19.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 5 2024-09-01T01:58:19.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-01T01:58:19.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:19.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:20.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:20.057 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:20.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:20.057 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:20.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:20.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:20.058 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:20.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:20.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:20.059 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:20.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:20.061 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:20.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:20.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:20.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:20.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_expect_failure td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_expect_failure: local dir=td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_expect_failure: setup td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:20.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:20.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:20.066 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:20.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:20.067 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:20.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:20.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:20.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:20.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:20.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:20.069 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:20.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:20.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:20.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:20.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.072 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:20.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:20.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:20.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:58:20.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:58:20.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:58:20.076 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:58:20.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:58:20.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:58:20.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-01T01:58:20.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:58:20.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:58:20.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:58:20.079 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-01T01:58:20.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: expect_failure: success=true 2024-09-01T01:58:20.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: true 2024-09-01T01:58:20.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: cat td/ceph-helpers/out 2024-09-01T01:58:20.080 INFO:tasks.workunit.client.0.smithi037.stderr:FAIL 2024-09-01T01:58:20.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: return 1 2024-09-01T01:58:20.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1997: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-01T01:58:20.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/ceph-helpers 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=FAIL 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-01T01:58:20.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T01:58:20.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T01:58:20.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-01T01:58:20.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: cat td/ceph-helpers/out 2024-09-01T01:58:20.084 INFO:tasks.workunit.client.0.smithi037.stderr:UNEXPECTED 2024-09-01T01:58:20.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: return 1 2024-09-01T01:58:20.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-01T01:58:20.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_expect_failure: teardown td/ceph-helpers 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:20.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:20.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:20.087 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:20.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:20.088 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:20.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:20.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:20.089 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:20.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:20.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:20.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:20.092 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:20.092 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:20.093 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:20.094 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.094 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.094 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: setup td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:58:20.096 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:58:20.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:58:20.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:58:20.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:58:20.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:58:20.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:58:20.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:58:20.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:58:20.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:58:20.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:58:20.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:58:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:58:20.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:58:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:58:20.100 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:58:20.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:58:20.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:58:20.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:58:20.103 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:58:20.103 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.103 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:58:20.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:58:20.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:58:20.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:58:20.105 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:58:20.105 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.105 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:58:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:58:20.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:58:20.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.545 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:20.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:58:20.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:20.730 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:20.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:58:20.816 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:58:20.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:58:20.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-01T01:58:20.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:58:20.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:58:20.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:58:20.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:58:20.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:58:20.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:58:21.752 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:21.753 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:58:21.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:58:21.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-01T01:58:21.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:58:21.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:21.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:58:21.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:58:21.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:58:21.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ddfa0611-f492-4224-b48d-1a51b2cca519 2024-09-01T01:58:21.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ddfa0611-f492-4224-b48d-1a51b2cca519' 2024-09-01T01:58:21.788 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 ddfa0611-f492-4224-b48d-1a51b2cca519 2024-09-01T01:58:21.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:58:21.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9ytNmVCzvLxAAFhew8CPuGCp4VS3R6fRV6A== 2024-09-01T01:58:21.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9ytNmVCzvLxAAFhew8CPuGCp4VS3R6fRV6A=="}' 2024-09-01T01:58:21.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ddfa0611-f492-4224-b48d-1a51b2cca519 -i td/ceph-helpers/0/new.json 2024-09-01T01:58:22.079 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:58:22.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:58:22.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9ytNmVCzvLxAAFhew8CPuGCp4VS3R6fRV6A== --osd-uuid ddfa0611-f492-4224-b48d-1a51b2cca519 2024-09-01T01:58:22.120 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:22.119+0000 7f5ca711e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:22.147 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:22.147+0000 7f5ca711e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:22.148 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:22.148+0000 7f5ca711e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:22.149 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:22.148+0000 7f5ca711e640 -1 bdev(0x55ce6b223c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:58:22.149 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:22.149+0000 7f5ca711e640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:58:28.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:58:28.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:58:28.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:58:28.077 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:58:28.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:58:28.654 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:58:28.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:58:28.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:58:28.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:58:28.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:58:28.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:58:28.681 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:28.681+0000 7f9c2d4ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:28.683 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:28.683+0000 7f9c2d4ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:28.684 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:28.684+0000 7f9c2d4ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:29.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:58:29.108 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:58:29.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:29.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:29.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:30.039 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:30.038+0000 7f9c2d4ed640 -1 Falling back to public interface 2024-09-01T01:58:30.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:30.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:30.542 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:58:30.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:58:30.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:30.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:30.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:31.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:31.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:31.994 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:58:31.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:58:31.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:31.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:32.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:33.072 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:58:33.071+0000 7f9c2d4ed640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:58:33.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:33.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:33.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:58:33.445 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:58:33.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:33.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:34.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:35.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:35.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:35.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:58:35.110 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:58:35.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:35.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:35.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:36.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:36.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:36.609 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:58:36.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:58:36.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:36.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:37.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:38.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:38.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:38.157 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:58:38.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:58:38.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:38.158 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:38.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:39.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:39.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:39.630 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:58:39.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:58:39.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:39.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:40.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:41.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:41.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:41.119 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:58:41.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:58:41.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:41.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:41.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:42.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:42.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:42.534 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:58:42.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:58:42.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:42.535 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:42.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:43.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:43.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:43.949 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:58:43.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:58:43.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:43.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:44.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:45.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:45.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:45.371 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:58:45.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:58:45.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:45.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:46.132 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:47.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:47.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:47.134 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:58:47.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:58:47.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:47.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:48.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:49.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:49.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:49.025 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:58:49.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:58:49.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:49.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:49.862 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:50.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:50.863 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:50.863 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:58:50.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:58:50.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:50.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:51.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:52.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:52.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:52.572 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:58:52.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:58:52.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:52.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:53.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:54.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:54.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:54.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:58:54.382 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:58:54.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:54.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:55.295 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:58:56.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:58:56.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:58:56.297 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:58:56.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:58:56.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:58:56.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:58:56.775 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/485158346,v1:127.0.0.1:6803/485158346] [v2:127.0.0.1:6804/485158346,v1:127.0.0.1:6805/485158346] exists,up ddfa0611-f492-4224-b48d-1a51b2cca519 2024-09-01T01:58:56.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:58:56.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:58:56.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:58:56.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_flush_pg_stats: create_rbd_pool 2024-09-01T01:58:56.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T01:58:57.199 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T01:58:57.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T01:58:57.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T01:58:57.704 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T01:58:57.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T01:58:58.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T01:58:59.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-01T01:58:59.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_flush_pg_stats: flush_pg_stats 2024-09-01T01:58:59.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T01:58:59.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T01:59:00.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T01:59:00.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T01:59:00.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T01:59:00.283 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-01T01:59:00.399 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T01:59:00.400 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T01:59:00.400 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-01T01:59:00.400 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T01:59:00.401 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-01T01:59:00.401 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836484 2024-09-01T01:59:00.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-01T01:59:00.402 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:59:00.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-01T01:59:00.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T01:59:01.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T01:59:01.807 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T01:59:02.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-01T01:59:02.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-01T01:59:02.277 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: ceph df detail --format=json 2024-09-01T01:59:02.277 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-01T01:59:02.831 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: stored=6680 2024-09-01T01:59:02.832 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: ceph df detail --format=json 2024-09-01T01:59:02.832 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-01T01:59:03.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: stored_raw=6680 2024-09-01T01:59:03.395 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_flush_pg_stats: test 6680 -gt 0 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_flush_pg_stats: test 6680 == 6680 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:03.396 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:03.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:03.500 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:03.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:03.501 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:03.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:03.502 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:03.502 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:03.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:03.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:03.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:03.503 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:03.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:03.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:03.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:03.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:03.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:03.511 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:03.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_config td/ceph-helpers 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:59:03.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:03.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:03.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:03.515 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:03.515 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:03.515 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:03.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:03.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:03.516 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:03.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:03.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:03.517 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:03.517 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:03.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:03.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:03.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:03.520 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:03.520 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:03.520 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:03.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:03.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:03.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:03.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:59:03.522 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:59:03.522 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:03.522 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:03.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:59:03.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:59:03.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:59:03.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:59:03.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:59:03.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:03.957 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:59:03.957 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:03.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:04.141 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:59:04.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:59:04.143 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:04.144 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:59:04.225 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:04.226 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:04.306 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:04.307 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:04.307 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:04.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:04.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get osd_pool_default_size 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:59:04.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:59:05.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:05.249 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:05.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-01T01:59:05.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:59:05.280 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:59:05.281 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:59:05.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:59:05.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=605f0c18-f4c8-4c1f-9956-cc86c964852b 2024-09-01T01:59:05.284 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 605f0c18-f4c8-4c1f-9956-cc86c964852b 2024-09-01T01:59:05.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 605f0c18-f4c8-4c1f-9956-cc86c964852b' 2024-09-01T01:59:05.284 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:59:05.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpytNmhRPZERAAKVPaIwRhIxcS4WXb1dRehw== 2024-09-01T01:59:05.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpytNmhRPZERAAKVPaIwRhIxcS4WXb1dRehw=="}' 2024-09-01T01:59:05.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 605f0c18-f4c8-4c1f-9956-cc86c964852b -i td/ceph-helpers/0/new.json 2024-09-01T01:59:05.591 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:59:05.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:59:05.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBpytNmhRPZERAAKVPaIwRhIxcS4WXb1dRehw== --osd-uuid 605f0c18-f4c8-4c1f-9956-cc86c964852b 2024-09-01T01:59:05.631 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:05.631+0000 7fcaed617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:05.660 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:05.659+0000 7fcaed617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:05.661 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:05.660+0000 7fcaed617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:05.661 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:05.661+0000 7fcaed617640 -1 bdev(0x55cfaf84dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:59:05.662 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:05.661+0000 7fcaed617640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:59:11.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:59:11.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:59:11.348 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:59:11.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:59:11.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:59:11.911 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:59:11.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:59:11.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-01T01:59:11.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:59:11.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:59:11.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:59:11.939 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:11.939+0000 7fc3bac0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:11.941 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:11.941+0000 7fc3bac0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:11.942 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:11.942+0000 7fc3bac0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:12.362 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:59:12.363 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:59:12.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:12.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:12.795 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:13.529 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:13.528+0000 7fc3bac0d640 -1 Falling back to public interface 2024-09-01T01:59:13.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:13.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:13.797 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:59:13.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:59:13.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:13.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:14.255 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:15.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:15.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:15.257 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:59:15.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:59:15.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:15.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:15.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:16.110 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:16.109+0000 7fc3bac0d640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:59:16.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:16.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:16.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:59:16.672 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:59:16.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:16.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:17.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:18.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:18.173 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:59:18.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:18.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:59:18.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:18.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:18.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:19.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:19.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:19.590 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:59:19.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:59:19.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:19.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:20.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:21.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:21.020 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:59:21.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:59:21.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:21.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:21.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:22.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:22.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:22.448 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T01:59:22.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T01:59:22.449 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:22.449 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:22.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:23.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:23.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:23.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T01:59:23.882 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T01:59:23.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:23.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:24.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:25.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:25.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:25.301 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T01:59:25.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T01:59:25.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:25.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:25.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:26.867 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:26.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:26.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T01:59:26.868 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T01:59:26.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:26.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:27.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:28.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:28.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:28.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T01:59:28.392 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T01:59:28.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:28.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:28.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:29.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:29.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:29.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T01:59:29.973 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T01:59:29.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:29.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:30.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:31.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:31.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:31.685 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T01:59:31.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T01:59:31.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:31.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:32.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:33.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:33.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:33.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T01:59:33.667 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T01:59:33.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:33.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:34.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:35.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:35.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:35.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T01:59:35.198 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T01:59:35.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:35.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:35.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:36.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:36.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:36.710 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T01:59:36.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T01:59:36.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:36.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:37.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:38.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:38.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:38.483 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T01:59:38.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T01:59:38.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:38.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:39.001 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3764485718,v1:127.0.0.1:6803/3764485718] [v2:127.0.0.1:6804/3764485718,v1:127.0.0.1:6805/3764485718] exists,up 605f0c18-f4c8-4c1f-9956-cc86c964852b 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T01:59:39.003 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:39.004 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.004 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.004 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T01:59:39.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:39.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_max_scrubs 2024-09-01T01:59:39.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:39.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:39.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:39.195 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:39.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:39.196 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:39.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:39.197 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:39.197 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:39.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:39.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:39.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:39.199 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:39.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:39.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:39.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:39.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:39.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:39.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:39.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:39.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:39.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:39.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:39.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:39.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:39.209 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:39.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:39.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:39.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:39.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:39.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:39.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:39.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:39.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:39.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:39.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:39.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:39.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:39.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:39.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:39.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:39.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:39.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:59:39.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:59:39.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.217 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:59:39.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T01:59:39.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:59:39.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:39.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T01:59:39.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:59:39.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:59:39.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:39.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:59:39.986 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:59:39.986 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:39.986 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:39.986 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:59:39.986 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:39.987 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:59:40.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-01T01:59:40.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:59:40.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:59:40.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:59:40.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:59:40.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:59:40.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:59:40.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:59:40.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:40.924 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:40.924 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:40.924 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:40.925 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:40.925 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:40.925 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:40.925 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:40.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-01T01:59:40.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T01:59:40.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T01:59:40.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T01:59:40.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T01:59:40.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T01:59:40.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T01:59:41.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T01:59:41.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T01:59:41.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-01T01:59:41.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:41.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:41.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:41.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:41.296 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:41.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:41.296 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:41.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:41.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:41.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:41.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:41.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:41.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:41.299 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:41.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:41.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:41.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:41.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:41.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:41.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:41.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:41.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T01:59:41.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T01:59:41.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T01:59:41.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T01:59:41.308 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T01:59:41.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:41.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T01:59:41.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T01:59:41.310 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T01:59:41.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T01:59:41.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T01:59:41.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T01:59:41.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T01:59:41.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:41.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:41.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T01:59:41.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T01:59:41.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T01:59:41.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T01:59:41.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T01:59:41.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:41.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:41.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T01:59:41.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T01:59:41.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:41.782 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:41.783 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:41.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T01:59:41.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T01:59:41.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T01:59:41.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:41.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:41.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:41.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T01:59:42.072 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T01:59:42.073 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T01:59:42.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-01T01:59:42.152 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T01:59:42.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T01:59:42.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T01:59:42.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T01:59:42.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T01:59:42.153 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T01:59:43.007 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T01:59:43.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:43.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:43.008 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:43.009 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:43.009 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:43.009 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:43.009 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:43.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T01:59:43.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:43.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T01:59:43.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T01:59:43.045 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bc426424-9895-4e12-9d08-c9f46e6a2d25 2024-09-01T01:59:43.045 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bc426424-9895-4e12-9d08-c9f46e6a2d25' 2024-09-01T01:59:43.045 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 bc426424-9895-4e12-9d08-c9f46e6a2d25 2024-09-01T01:59:43.046 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T01:59:43.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCPytNmLYCJAxAAnA/huEVcVJZ1+bSB2PHOdQ== 2024-09-01T01:59:43.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCPytNmLYCJAxAAnA/huEVcVJZ1+bSB2PHOdQ=="}' 2024-09-01T01:59:43.060 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bc426424-9895-4e12-9d08-c9f46e6a2d25 -i td/ceph-helpers/0/new.json 2024-09-01T01:59:43.349 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:59:43.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T01:59:43.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPytNmLYCJAxAAnA/huEVcVJZ1+bSB2PHOdQ== --osd-uuid bc426424-9895-4e12-9d08-c9f46e6a2d25 2024-09-01T01:59:43.389 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:43.389+0000 7f55867b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:43.417+0000 7f55867b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:43.418+0000 7f55867b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:43.418+0000 7f55867b9640 -1 bdev(0x55af03e23c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T01:59:43.421 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:43.419+0000 7f55867b9640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T01:59:49.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T01:59:49.142 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T01:59:49.142 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T01:59:49.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T01:59:49.143 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T01:59:49.710 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T01:59:49.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T01:59:49.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T01:59:49.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T01:59:49.711 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T01:59:49.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T01:59:49.737 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:49.736+0000 7f146fab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:49.738 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:49.738+0000 7f146fab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:49.739 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:49.739+0000 7f146fab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T01:59:50.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:50.162 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T01:59:50.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T01:59:50.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:50.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:50.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:50.835 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:50.835+0000 7f146fab0640 -1 Falling back to public interface 2024-09-01T01:59:51.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:51.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:51.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T01:59:51.588 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T01:59:51.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:51.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:52.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:53.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:53.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:53.031 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T01:59:53.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T01:59:53.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:53.032 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:53.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:53.892 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T01:59:53.891+0000 7f146fab0640 -1 osd.0 0 log_to_monitors true 2024-09-01T01:59:54.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:54.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:54.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T01:59:54.447 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T01:59:54.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:54.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:54.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:55.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:55.974 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:55.974 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T01:59:55.975 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T01:59:55.975 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:55.975 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:56.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:57.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:57.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:57.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T01:59:57.393 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T01:59:57.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:57.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:57.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T01:59:58.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T01:59:58.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T01:59:58.825 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T01:59:58.826 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T01:59:58.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T01:59:58.826 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T01:59:59.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:00.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:00.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:00.242 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:00:00.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:00:00.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:00.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:00.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:01.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:01.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:01.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:00:01.737 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:00:01.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:01.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:02.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:03.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:03.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:03.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:00:03.176 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:00:03.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:03.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:03.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:04.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:04.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:04.625 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:00:04.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:00:04.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:04.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:05.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:06.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:06.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:06.042 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:00:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:00:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:06.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:07.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:07.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:07.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:00:07.558 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:00:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:07.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:08.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:09.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:09.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:09.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:00:09.328 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:00:09.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:09.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:10.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:11.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:11.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:11.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:00:11.467 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:00:11.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:11.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:12.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:13.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:13.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:13.185 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:00:13.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:00:13.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:13.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:14.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:15.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:15.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:15.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:00:15.101 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:00:15.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:15.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:15.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:16.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:16.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:16.811 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:00:16.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:00:16.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:16.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:17.235 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3649469712,v1:127.0.0.1:6803/3649469712] [v2:127.0.0.1:6804/3649469712,v1:127.0.0.1:6805/3649469712] exists,up bc426424-9895-4e12-9d08-c9f46e6a2d25 2024-09-01T02:00:17.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:00:17.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:00:17.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:00:17.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2024-09-01T02:00:17.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:00:17.660 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:00:17.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:00:17.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:00:18.179 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:00:18.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:00:19.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:00:20.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2024-09-01T02:00:20.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:00:20.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:00:20.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:00:20.232 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:00:20.233 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:00:20.233 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:00:20.233 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:00:20.233 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:00:20.233 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:00:20.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:00:20.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:00:20.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:00:20.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:00:20.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:00:20.329 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:00:20.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:00:20.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:00:20.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:00:20.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:00:20.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-01T02:00:20.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-01T02:00:20.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-01T02:00:20.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:00:20.874 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-01T02:00:20.874 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:00:20.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:00:20.874 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-01T02:00:20.875 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:00:20.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-01T02:00:20.876 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836489 2024-09-01T02:00:20.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-01T02:00:20.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:00:21.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-01T02:00:21.279 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:00:21.279 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:00:21.279 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:00:21.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:00:21.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:00:21.836 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:00:21.836 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:00:21.836 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:00:21.836 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:00:21.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:00:21.837 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:00:22.255 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:00:22.256 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:00:22.256 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:00:22.256 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:00:22.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:00:23.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2024-09-01T02:00:18.002828+0000 2024-09-01T02:00:23.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2024-09-01T02:00:18.002828+0000 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:00:23.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:00:23.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:00:23.352 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:00:23.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:00:23.353 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:00:23.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:00:23.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:00:23.355 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:00:23.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:00:23.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:00:23.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:00:23.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:00:23.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:00:23.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:00:23.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:00:23.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:00:23.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:23.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:23.363 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_not_primary td/ceph-helpers 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:00:23.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:00:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:00:23.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:00:23.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:00:23.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:00:23.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:00:23.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:00:23.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:00:23.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:00:23.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:00:23.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:00:23.371 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:00:23.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:00:23.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:00:23.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:00:23.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:00:23.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:23.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:23.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:00:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:00:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:00:23.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:00:23.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:00:23.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:23.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:23.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:00:23.376 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:00:23.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:00:23.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:00:23.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:00:23.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:00:23.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:23.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-01T02:00:24.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:00:24.028 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:00:24.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:00:24.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:00:24.029 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:00:24.029 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:24.030 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:24.030 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:00:24.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:00:24.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:00:24.113 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:00:24.114 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:00:24.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-01T02:00:24.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:00:24.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:00:24.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:00:24.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:00:24.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:00:24.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:25.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:25.053 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:00:25.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:00:25.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:25.085 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:00:25.086 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7f7d9afa-8037-455b-a57e-8bac653780da 2024-09-01T02:00:25.087 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 7f7d9afa-8037-455b-a57e-8bac653780da 2024-09-01T02:00:25.087 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7f7d9afa-8037-455b-a57e-8bac653780da' 2024-09-01T02:00:25.088 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:00:25.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC5ytNmdOYcBhAAcj6LyZahKjMJsL/ckZ2eUA== 2024-09-01T02:00:25.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC5ytNmdOYcBhAAcj6LyZahKjMJsL/ckZ2eUA=="}' 2024-09-01T02:00:25.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7f7d9afa-8037-455b-a57e-8bac653780da -i td/ceph-helpers/0/new.json 2024-09-01T02:00:25.377 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:00:25.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:00:25.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5ytNmdOYcBhAAcj6LyZahKjMJsL/ckZ2eUA== --osd-uuid 7f7d9afa-8037-455b-a57e-8bac653780da 2024-09-01T02:00:25.417 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:25.415+0000 7f6440219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:25.448 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:25.445+0000 7f6440219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:25.449 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:25.446+0000 7f6440219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:25.449 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:25.446+0000 7f6440219640 -1 bdev(0x5615cc9e5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:00:25.449 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:25.446+0000 7f6440219640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:00:31.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:00:31.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:00:31.634 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:00:31.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:00:31.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:00:32.212 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:00:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:00:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:00:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:00:32.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:00:32.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:00:32.240 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:32.239+0000 7efe6ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:32.242 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:32.241+0000 7efe6ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:32.243 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:32.242+0000 7efe6ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:32.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:32.648 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:00:32.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:00:32.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:32.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:33.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:33.327 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:33.327+0000 7efe6ddcc640 -1 Falling back to public interface 2024-09-01T02:00:34.092 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:34.092 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:34.092 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:00:34.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:00:34.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:34.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:34.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:35.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:35.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:35.559 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:00:35.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:00:35.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:35.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:35.736 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:35.735+0000 7efe6ddcc640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:00:36.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:37.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:37.004 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:00:37.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:37.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:00:37.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:37.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:37.478 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:38.479 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:38.479 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:38.480 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:00:38.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:00:38.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:38.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:38.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:39.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:39.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:39.912 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:00:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:00:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:39.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:40.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:41.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:41.328 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:41.328 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:00:41.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:00:41.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:41.329 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:41.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:42.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:42.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:42.819 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:00:42.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:00:42.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:42.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:43.260 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:44.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:44.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:44.262 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:00:44.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:00:44.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:44.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:44.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:45.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:45.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:45.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:00:45.820 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:00:45.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:45.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:46.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:47.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:47.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:47.238 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:00:47.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:00:47.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:47.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:48.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:49.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:49.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:49.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:00:49.250 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:00:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:49.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:50.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:51.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:51.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:51.039 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:00:51.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:00:51.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:51.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:51.461 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:52.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:52.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:52.463 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:00:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:00:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:53.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:54.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:54.057 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:54.057 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:00:54.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:00:54.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:54.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:54.617 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:55.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:55.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:55.618 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:00:55.619 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:00:55.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:55.619 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:56.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:00:57.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:00:57.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:00:57.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:00:57.731 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:00:57.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:00:57.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:00:58.216 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4210979986,v1:127.0.0.1:6803/4210979986] [v2:127.0.0.1:6804/4210979986,v1:127.0.0.1:6805/4210979986] exists,up 7f7d9afa-8037-455b-a57e-8bac653780da 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:00:58.220 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:00:58.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:00:58.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-01T02:00:58.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:00:58.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5fcdb379-facf-4379-ae97-b8a49190f548 2024-09-01T02:00:58.223 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 5fcdb379-facf-4379-ae97-b8a49190f548 2024-09-01T02:00:58.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5fcdb379-facf-4379-ae97-b8a49190f548' 2024-09-01T02:00:58.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:00:58.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDaytNmfCYrDhAAutZwoOzMhUVYnJWnyMnNYg== 2024-09-01T02:00:58.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDaytNmfCYrDhAAutZwoOzMhUVYnJWnyMnNYg=="}' 2024-09-01T02:00:58.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5fcdb379-facf-4379-ae97-b8a49190f548 -i td/ceph-helpers/1/new.json 2024-09-01T02:00:58.768 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:00:58.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-01T02:00:58.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaytNmfCYrDhAAutZwoOzMhUVYnJWnyMnNYg== --osd-uuid 5fcdb379-facf-4379-ae97-b8a49190f548 2024-09-01T02:00:58.809 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:58.809+0000 7f559ee22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:58.811 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:58.810+0000 7f559ee22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:58.812 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:58.811+0000 7f559ee22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:00:58.812 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:58.812+0000 7f559ee22640 -1 bdev(0x55ed34b25c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:00:58.812 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:00:58.812+0000 7f559ee22640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-01T02:01:04.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-01T02:01:04.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:01:04.562 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T02:01:04.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:01:04.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:01:05.136 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T02:01:05.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:01:05.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:01:05.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:01:05.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:01:05.138 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:01:05.163 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:05.161+0000 7f6f58c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:05.166 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:05.164+0000 7f6f58c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:05.167 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:05.166+0000 7f6f58c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:05.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:01:05.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:01:05.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:01:05.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:01:05.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:01:05.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:05.558 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:01:05.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:01:05.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:05.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:05.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:06.508 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:06.507+0000 7f6f58c1a640 -1 Falling back to public interface 2024-09-01T02:01:06.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:06.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:06.991 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:01:06.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:01:06.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:06.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:07.425 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:08.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:08.426 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:08.427 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:01:08.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:01:08.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:08.427 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:08.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:09.405 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:09.404+0000 7f6f58c1a640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:01:09.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:09.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:09.839 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:01:09.839 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:01:09.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:09.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:10.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:11.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:11.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:11.319 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:01:11.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:01:11.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:11.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:11.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:12.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:12.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:12.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:01:12.749 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:01:12.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:12.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:13.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:14.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:14.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:14.164 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:01:14.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:01:14.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:14.164 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:14.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:15.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:15.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:15.678 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:01:15.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:01:15.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:15.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:16.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:17.108 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:01:17.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:01:17.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:17.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:17.653 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:18.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:18.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:18.654 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:01:18.654 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:01:18.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:18.655 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:19.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:20.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:20.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:20.108 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:01:20.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:01:20.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:20.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:20.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:21.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:21.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:21.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:01:21.663 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:01:21.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:21.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:22.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:23.238 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:23.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:23.239 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:01:23.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:01:23.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:23.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:23.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:24.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:24.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:24.722 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:01:24.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:01:24.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:24.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:26.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:27.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:27.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:27.112 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:01:27.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:01:27.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:27.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:28.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:29.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:29.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:29.113 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:01:29.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:01:29.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:29.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:30.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:31.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:31.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:31.118 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:01:31.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:01:31.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:31.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:31.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:32.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:32.858 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:01:32.858 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:32.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:01:32.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:32.859 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:01:33.291 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3533840421,v1:127.0.0.1:6811/3533840421] [v2:127.0.0.1:6812/3533840421,v1:127.0.0.1:6813/3533840421] exists,up 5fcdb379-facf-4379-ae97-b8a49190f548 2024-09-01T02:01:33.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:01:33.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:01:33.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:01:33.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2024-09-01T02:01:33.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:01:33.715 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:01:33.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:01:33.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:01:34.228 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:01:34.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:01:35.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:01:37.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2024-09-01T02:01:37.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:01:37.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:01:37.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:01:37.109 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:01:37.110 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:01:37.110 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:01:37.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:01:37.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:01:37.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:01:37.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:01:37.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:01:37.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:01:37.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:01:37.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:01:37.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:01:37.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:01:37.645 INFO:tasks.workunit.client.0.smithi037.stderr:1' 2024-09-01T02:01:37.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:01:37.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:01:37.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:01:37.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-01T02:01:37.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-01T02:01:37.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-01T02:01:37.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:01:37.761 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:01:37.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-01T02:01:37.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-01T02:01:37.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672969' 2024-09-01T02:01:37.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:01:37.879 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-01T02:01:37.879 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:01:37.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:01:37.881 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-01T02:01:37.882 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:01:37.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-01T02:01:37.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-01T02:01:37.882 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836493 2024-09-01T02:01:37.883 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:01:38.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836493 2024-09-01T02:01:38.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:01:39.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:01:39.300 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:01:39.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-01T02:01:39.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:01:39.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-01T02:01:39.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:01:39.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T02:01:39.736 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-01T02:01:39.736 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:01:39.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-01T02:01:39.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-01T02:01:39.737 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672969 2024-09-01T02:01:39.737 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:01:40.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-01T02:01:40.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:01:40.215 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:01:40.216 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:01:40.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:01:40.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:01:40.778 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:01:40.778 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:01:40.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:01:40.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:01:40.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:01:40.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:01:41.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:01:41.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:01:41.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:01:41.211 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:01:41.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-01T02:01:41.756 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-01T02:01:42.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2024-09-01T02:01:42.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2024-09-01T02:01:42.221 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-01T02:01:42.222 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-01T02:01:42.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-01T02:01:42.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-01T02:01:42.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-01T02:01:43.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:01:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:01:43.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:01:43.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:01:43.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:01:43.205 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:01:43.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:01:43.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:01:43.207 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:01:43.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:01:43.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:01:43.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:01:43.208 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:01:43.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:01:43.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:01:43.210 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:01:43.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:01:43.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.219 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.219 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:01:43.221 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:01:43.222 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:01:43.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:01:43.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:01:43.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:01:43.223 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:01:43.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:01:43.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:01:43.224 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:01:43.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:01:43.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:01:43.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:01:43.226 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:01:43.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:01:43.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:01:43.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:01:43.228 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:01:43.228 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.229 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:01:43.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:01:43.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:01:43.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:01:43.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:01:43.230 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.231 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:01:43.232 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:01:43.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:01:43.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:01:43.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:01:43.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:01:43.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:01:43.895 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:01:43.896 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:01:43.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:01:43.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:01:43.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:01:43.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:01:43.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:01:44.058 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:01:44.059 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:01:44.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:01:44.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:01:44.917 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:01:44.917 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:01:44.917 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:44.918 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:44.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:01:44.918 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:01:44.918 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:01:44.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:01:44.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:01:44.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71dc85a0-5193-4c63-a412-83f077cfa0a6 2024-09-01T02:01:44.954 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 71dc85a0-5193-4c63-a412-83f077cfa0a6 2024-09-01T02:01:44.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 71dc85a0-5193-4c63-a412-83f077cfa0a6' 2024-09-01T02:01:44.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:01:44.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIy9Nmw1u3ORAAk7YI4Fsbw/zS3DSFIvyXYA== 2024-09-01T02:01:44.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIy9Nmw1u3ORAAk7YI4Fsbw/zS3DSFIvyXYA=="}' 2024-09-01T02:01:44.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71dc85a0-5193-4c63-a412-83f077cfa0a6 -i td/ceph-helpers/0/new.json 2024-09-01T02:01:45.244 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:01:45.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:01:45.254 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIy9Nmw1u3ORAAk7YI4Fsbw/zS3DSFIvyXYA== --osd-uuid 71dc85a0-5193-4c63-a412-83f077cfa0a6 2024-09-01T02:01:45.283 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:45.282+0000 7f2c53e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:45.329 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:45.328+0000 7f2c53e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:45.330 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:45.329+0000 7f2c53e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:45.330 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:45.329+0000 7f2c53e18640 -1 bdev(0x55a012f33c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:01:45.330 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:45.330+0000 7f2c53e18640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:01:51.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:01:51.365 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:01:51.365 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:01:51.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:01:51.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:01:51.938 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:01:51.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:01:51.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:01:51.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:01:51.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:01:51.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:01:51.965 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:51.964+0000 7f419c754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:51.967 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:51.966+0000 7f419c754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:51.968 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:51.967+0000 7f419c754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:01:52.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:01:52.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:01:52.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:01:52.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:01:52.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:01:52.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:52.380 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:01:52.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:01:52.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:52.381 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:01:52.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:53.068 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:53.068+0000 7f419c754640 -1 Falling back to public interface 2024-09-01T02:01:53.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:53.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:53.815 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:01:53.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:01:53.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:53.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:01:54.255 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:55.256 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:55.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:55.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:01:55.257 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:01:55.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:55.258 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:01:55.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:56.119 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:01:56.118+0000 7f419c754640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:01:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:01:56.678 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:01:56.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:56.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:01:57.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:58.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:58.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:58.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:01:58.223 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:01:58.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:58.224 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:01:58.659 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:01:59.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:01:59.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:01:59.661 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:01:59.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:01:59.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:01:59.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:00.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:01.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:01.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:01.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:02:01.195 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:02:01.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:01.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:01.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:02.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:02.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:02.624 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:02:02.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:02:02.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:02.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:03.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:04.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:04.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:04.043 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:02:04.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:02:04.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:04.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:04.457 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:05.459 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:02:05.459 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:05.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:02:05.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:05.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:05.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:06.882 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:06.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:06.883 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:02:06.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:02:06.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:06.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:07.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:08.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:08.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:08.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:02:08.301 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:02:08.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:08.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:08.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:09.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:09.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:09.891 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:02:09.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:02:09.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:09.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:10.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:11.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:11.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:11.777 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:02:11.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:02:11.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:11.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:12.838 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:13.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:13.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:13.840 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:02:13.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:02:13.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:13.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:14.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:15.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:15.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:15.510 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:02:15.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:02:15.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:15.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:16.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:17.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:17.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:17.423 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:02:17.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:02:17.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:17.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:18.243 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:19.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:19.244 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:19.245 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:02:19.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:02:19.245 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:19.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:19.693 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2856668063,v1:127.0.0.1:6803/2856668063] [v2:127.0.0.1:6804/2856668063,v1:127.0.0.1:6805/2856668063] exists,up 71dc85a0-5193-4c63-a412-83f077cfa0a6 2024-09-01T02:02:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:02:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:02:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:02:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2024-09-01T02:02:19.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:02:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:02:20.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:02:20.111 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:02:20.613 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:02:20.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:02:21.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:02:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2024-09-01T02:02:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:02:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:02:22.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:02:22.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:02:22.648 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:02:22.648 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:02:22.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:02:22.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:02:22.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:02:22.742 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:02:23.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:02:23.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:02:23.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:02:23.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:02:23.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:02:23.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:02:23.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-01T02:02:23.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:02:23.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-01T02:02:23.369 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836488 2024-09-01T02:02:23.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-01T02:02:23.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:02:23.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-01T02:02:23.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:02:24.831 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:02:24.832 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:02:25.257 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-01T02:02:25.258 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:02:25.258 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:02:25.258 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:02:25.817 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:02:26.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:02:26.227 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:02:26.227 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:02:26.227 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:02:26.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:02:26.779 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:02:27.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2024-09-01T02:02:27.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:02:27.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:02:27.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:02:27.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:02:27.299 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:02:27.299 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:02:27.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:02:27.300 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:02:27.300 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:02:27.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:02:27.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:02:27.301 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:02:27.302 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:02:27.302 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:02:27.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:02:27.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:02:27.309 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:02:27.309 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:27.309 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:27.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:02:27.311 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:02:27.312 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:02:27.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:02:27.314 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:02:27.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:02:27.315 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:02:27.315 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:02:27.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:02:27.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:02:27.316 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:02:27.316 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:02:27.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:02:27.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:02:27.318 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:02:27.319 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:02:27.319 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:27.319 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:27.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:02:27.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:02:27.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:02:27.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:02:27.321 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:02:27.321 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:27.321 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:27.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:02:27.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:02:27.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:02:27.327 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:02:27.775 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:02:27.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:02:27.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:02:27.969 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:02:27.970 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:02:27.970 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:27.970 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:27.970 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:02:27.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:02:27.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:02:28.055 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:02:28.056 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:02:28.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-01T02:02:28.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:02:28.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:02:28.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:02:28.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:02:28.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:02:28.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:02:28.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:02:29.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-01T02:02:29.025 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:02:29.026 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:02:29.027 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:02:29.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3498f5ec-fb76-4e82-a967-8a14758b6109 2024-09-01T02:02:29.029 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 3498f5ec-fb76-4e82-a967-8a14758b6109 2024-09-01T02:02:29.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3498f5ec-fb76-4e82-a967-8a14758b6109' 2024-09-01T02:02:29.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:02:29.045 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1y9NmuoqtAhAAFQhpkaOeGxrwjwFfXMd2Fg== 2024-09-01T02:02:29.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1y9NmuoqtAhAAFQhpkaOeGxrwjwFfXMd2Fg=="}' 2024-09-01T02:02:29.046 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3498f5ec-fb76-4e82-a967-8a14758b6109 -i td/ceph-helpers/0/new.json 2024-09-01T02:02:29.327 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:02:29.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:02:29.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1y9NmuoqtAhAAFQhpkaOeGxrwjwFfXMd2Fg== --osd-uuid 3498f5ec-fb76-4e82-a967-8a14758b6109 2024-09-01T02:02:29.367 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:29.366+0000 7f42a962c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:29.394 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:29.393+0000 7f42a962c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:29.396 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:29.395+0000 7f42a962c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:29.396 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:29.395+0000 7f42a962c640 -1 bdev(0x55bb762cdc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:02:29.396 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:29.395+0000 7f42a962c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:02:35.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:02:35.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:02:35.433 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:02:35.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:02:35.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:02:36.010 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:02:36.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:02:36.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:02:36.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:02:36.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:02:36.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:02:36.038 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:36.037+0000 7fb2849cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:36.039 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:36.039+0000 7fb2849cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:36.041 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:36.040+0000 7fb2849cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:02:36.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:02:36.448 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:02:36.449 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:36.449 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:36.877 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:36.876+0000 7fb2849cc640 -1 Falling back to public interface 2024-09-01T02:02:36.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:37.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:37.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:37.885 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:02:37.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:02:37.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:37.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:38.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:39.286 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:02:39.286+0000 7fb2849cc640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:02:39.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:39.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:39.325 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:02:39.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:02:39.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:39.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:39.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:40.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:40.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:40.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:02:40.991 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:02:40.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:40.991 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:41.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:42.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:42.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:42.440 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:02:42.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:02:42.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:42.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:42.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:43.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:43.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:43.883 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:02:43.883 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:02:43.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:43.884 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:44.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:45.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:45.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:45.297 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:02:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:02:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:45.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:45.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:46.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:46.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:46.729 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:02:46.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:02:46.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:46.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:47.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:48.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:48.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:48.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:02:48.231 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:02:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:48.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:02:49.647 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:02:49.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:49.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:50.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:51.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:51.095 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:51.095 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:02:51.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:02:51.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:51.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:51.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:52.831 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:52.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:52.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:02:52.832 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:02:52.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:52.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:53.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:54.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:54.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:54.643 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:02:54.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:02:54.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:54.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:55.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:56.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:56.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:56.267 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:02:56.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:02:56.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:56.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:56.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:57.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:57.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:57.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:02:57.936 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:02:57.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:57.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:02:58.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:02:59.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:02:59.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:02:59.692 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:02:59.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:02:59.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:02:59.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:00.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:01.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:01.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:01.663 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:03:01.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:03:01.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:01.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:02.225 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1128569204,v1:127.0.0.1:6803/1128569204] [v2:127.0.0.1:6804/1128569204,v1:127.0.0.1:6805/1128569204] exists,up 3498f5ec-fb76-4e82-a967-8a14758b6109 2024-09-01T02:03:02.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:03:02.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:03:02.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:03:02.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2024-09-01T02:03:02.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:03:02.772 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:03:02.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:03:02.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:03:03.297 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:03:03.308 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:03:04.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:03:05.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2024-09-01T02:03:05.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:03:05.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:03:05.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:03:05.388 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:03:05.389 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:03:05.389 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:03:05.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:03:05.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:03:05.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:03:05.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:03:05.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:03:05.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:03:05.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:03:05.484 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:03:05.484 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:03:05.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:03:05.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:03:05.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:03:05.923 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:03:06.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-01T02:03:06.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-01T02:03:06.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-01T02:03:06.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:03:06.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-01T02:03:06.040 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:03:06.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:03:06.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-01T02:03:06.041 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:03:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-01T02:03:06.042 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836485 2024-09-01T02:03:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-01T02:03:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:03:06.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-01T02:03:06.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:03:07.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:03:07.501 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:03:07.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-01T02:03:07.959 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:03:07.959 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:03:07.959 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:03:08.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:03:08.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:03:08.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:03:08.979 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:03:08.980 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:03:09.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-01T02:03:09.534 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:03:09.946 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:03:09.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:03:09.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:03:09.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:03:10.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:03:10.051 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:03:10.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:03:10.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:03:10.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:03:10.053 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:03:10.054 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:03:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:03:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:03:10.055 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:10.055 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:03:10.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:03:10.056 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:03:10.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:03:10.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.062 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:03:10.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:03:10.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:03:10.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:03:10.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:03:10.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:03:10.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:03:10.066 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:03:10.066 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:03:10.067 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:03:10.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:03:10.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:03:10.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:03:10.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:10.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:03:10.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:03:10.069 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:10.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:03:10.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:03:10.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:03:10.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:03:10.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.072 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:03:10.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:03:10.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:03:10.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:03:10.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:03:10.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.074 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:03:10.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:03:10.077 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:10.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:03:10.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.745 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.746 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:03:10.746 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:03:10.746 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:03:10.828 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:03:10.829 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:03:10.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:11.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:03:11.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:03:11.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:11.802 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:03:11.803 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:03:11.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:03:11.804 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:03:11.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=62bc72fe-bc87-4f51-81a0-ab4b46501105 2024-09-01T02:03:11.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 62bc72fe-bc87-4f51-81a0-ab4b46501105' 2024-09-01T02:03:11.805 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 62bc72fe-bc87-4f51-81a0-ab4b46501105 2024-09-01T02:03:11.805 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:03:11.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBfy9Nmw6LiMBAAz6sA1pL6POSyywDpUoX1SA== 2024-09-01T02:03:11.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBfy9Nmw6LiMBAAz6sA1pL6POSyywDpUoX1SA=="}' 2024-09-01T02:03:11.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 62bc72fe-bc87-4f51-81a0-ab4b46501105 -i td/ceph-helpers/0/new.json 2024-09-01T02:03:12.109 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:03:12.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:03:12.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfy9Nmw6LiMBAAz6sA1pL6POSyywDpUoX1SA== --osd-uuid 62bc72fe-bc87-4f51-81a0-ab4b46501105 2024-09-01T02:03:12.149 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:12.148+0000 7fa0a8502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:12.178 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:12.177+0000 7fa0a8502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:12.179 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:12.178+0000 7fa0a8502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:12.179 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:12.179+0000 7fa0a8502640 -1 bdev(0x55e0dd8dbc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:03:12.179 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:12.179+0000 7fa0a8502640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:03:18.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:03:18.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:03:18.850 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:03:18.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:03:18.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:03:19.419 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:03:19.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:03:19.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:03:19.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:03:19.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:03:19.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:03:19.446 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:19.445+0000 7ff1d9789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:19.448 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:19.447+0000 7ff1d9789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:19.449 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:19.448+0000 7ff1d9789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:19.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:03:19.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:03:19.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:03:19.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:03:19.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:03:19.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:19.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:03:19.871 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:03:19.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:19.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:20.296 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:20.783 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:20.782+0000 7ff1d9789640 -1 Falling back to public interface 2024-09-01T02:03:21.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:21.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:21.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:03:21.298 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:03:21.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:21.298 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:21.712 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:22.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:22.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:22.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:03:22.714 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:03:22.714 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:22.715 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:23.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:23.818 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:23.816+0000 7ff1d9789640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:03:24.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:24.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:24.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:03:24.183 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:03:24.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:24.183 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:24.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:25.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:25.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:25.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:03:25.698 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:03:25.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:25.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:26.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:27.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:27.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:27.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:03:27.116 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:03:27.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:27.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:27.573 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:28.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:28.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:28.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:03:28.575 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:03:28.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:28.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:28.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:29.996 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:29.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:29.997 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:03:29.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:03:29.997 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:29.998 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:30.578 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:31.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:31.579 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:31.579 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:03:31.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:03:31.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:31.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:31.998 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:32.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:32.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:32.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:03:32.999 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:03:33.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:33.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:33.574 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:34.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:34.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:34.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:03:34.576 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:03:34.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:34.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:34.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:35.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:35.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:35.992 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:03:35.992 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:03:35.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:35.993 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:37.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:38.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:38.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:38.268 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:03:38.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:03:38.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:38.269 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:38.767 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:39.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:39.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:39.769 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:03:39.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:03:39.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:39.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:40.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:41.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:41.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:41.184 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:03:41.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:03:41.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:41.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:41.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:42.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:42.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:42.982 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:03:42.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:03:42.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:42.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:43.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:44.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:44.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:44.590 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:03:44.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:03:44.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:44.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:45.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:03:46.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:03:46.640 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:03:46.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:03:46.641 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:03:46.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:03:46.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:03:47.106 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1298075847,v1:127.0.0.1:6803/1298075847] [v2:127.0.0.1:6804/1298075847,v1:127.0.0.1:6805/1298075847] exists,up 62bc72fe-bc87-4f51-81a0-ab4b46501105 2024-09-01T02:03:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:03:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:03:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:03:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2024-09-01T02:03:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:03:47.498 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:03:47.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:03:47.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:03:48.010 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:03:48.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:03:49.023 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:03:49.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2024-09-01T02:03:49.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:03:49.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:03:49.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:03:49.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:03:49.984 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:03:49.984 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:03:49.984 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:03:49.984 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:03:49.984 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:03:50.078 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:03:50.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:03:50.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:03:50.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:03:50.513 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:03:50.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-01T02:03:50.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-01T02:03:50.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-01T02:03:50.630 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:03:50.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-01T02:03:50.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:03:50.631 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:03:50.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-01T02:03:50.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:03:50.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-01T02:03:50.632 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836489 2024-09-01T02:03:50.633 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-01T02:03:50.633 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:03:51.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-01T02:03:51.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:03:52.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:03:52.034 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:03:52.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-01T02:03:52.482 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:03:52.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:03:52.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:03:53.034 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:03:53.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:03:53.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:03:53.430 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:03:53.430 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:03:53.430 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:03:53.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:03:53.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:03:53.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:03:53.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2024-09-01T02:03:53.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:03:53.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:03:54.564 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:03:54.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:03:54.670 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:03:54.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:03:54.670 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:03:54.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:03:54.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:03:54.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:03:54.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:54.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:03:54.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:03:54.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:54.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:03:54.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:03:54.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:03:54.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:03:54.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:54.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:54.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: test_get_op_scheduler: setup td/ceph-helpers 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:03:54.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:03:54.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:03:54.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:03:54.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:03:54.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:03:54.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:03:54.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:03:54.686 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:03:54.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:54.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:03:54.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:03:54.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:03:54.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:03:54.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:03:54.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:03:54.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:03:54.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:54.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:54.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:03:54.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:03:54.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:03:54.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:03:54.692 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:03:54.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:54.693 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:54.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:03:54.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:03:54.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:55.166 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:55.167 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:55.167 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T02:03:55.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:03:55.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:03:55.367 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:03:55.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:03:55.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:55.369 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:03:55.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:03:55.370 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:03:55.451 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:03:55.451 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:03:55.451 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:55.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:03:55.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:03:55.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:03:55.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-01T02:03:55.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:03:55.531 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:03:55.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:03:55.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:03:55.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:03:55.532 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:03:56.382 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:03:56.382 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:03:56.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:03:56.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:03:56.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:03:56.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:03:56.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43cb68e1-ee6d-46de-8541-99fccbac803b 2024-09-01T02:03:56.419 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 43cb68e1-ee6d-46de-8541-99fccbac803b 2024-09-01T02:03:56.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 43cb68e1-ee6d-46de-8541-99fccbac803b' 2024-09-01T02:03:56.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:03:56.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMy9NmDbDTGRAALL2tvpSKMdM6GlEGeo0sDA== 2024-09-01T02:03:56.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMy9NmDbDTGRAALL2tvpSKMdM6GlEGeo0sDA=="}' 2024-09-01T02:03:56.434 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43cb68e1-ee6d-46de-8541-99fccbac803b -i td/ceph-helpers/0/new.json 2024-09-01T02:03:56.716 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:03:56.726 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:03:56.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCMy9NmDbDTGRAALL2tvpSKMdM6GlEGeo0sDA== --osd-uuid 43cb68e1-ee6d-46de-8541-99fccbac803b 2024-09-01T02:03:56.757 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:56.756+0000 7ff330720640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:56.785 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:56.784+0000 7ff330720640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:56.786 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:56.785+0000 7ff330720640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:03:56.786 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:56.786+0000 7ff330720640 -1 bdev(0x562d5b219c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:03:56.787 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:03:56.786+0000 7ff330720640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:04:03.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:04:03.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:04:03.415 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:04:03.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:04:03.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:04:03.977 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:04:03.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:04:03.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-01T02:04:03.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:04:03.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:04:03.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:04:04.005 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:04.004+0000 7fbc7b00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:04.006 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:04.006+0000 7fbc7b00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:04.007 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:04.007+0000 7fbc7b00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:04.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:04:04.437 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:04:04.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:04.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:04.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:05.344 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:05.343+0000 7fbc7b00b640 -1 Falling back to public interface 2024-09-01T02:04:06.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:06.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:06.146 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:04:06.146 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:04:06.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:06.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:06.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:07.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:07.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:07.324 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:04:07.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:04:07.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:07.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:07.789 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:08.312 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:08.309+0000 7fbc7b00b640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:04:08.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:08.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:08.791 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:04:08.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:04:08.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:08.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:09.322 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2584235560,v1:127.0.0.1:6803/2584235560] [v2:127.0.0.1:6804/2584235560,v1:127.0.0.1:6805/2584235560] exists,up 43cb68e1-ee6d-46de-8541-99fccbac803b 2024-09-01T02:04:09.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: get_op_scheduler 0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: get_op_scheduler: local id=0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T02:04:09.325 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:04:09.326 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:09.326 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:09.326 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.0.asok 2024-09-01T02:04:09.326 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:04:09.326 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-osd.0.asok config get osd_op_queue 2024-09-01T02:04:09.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: test wpq = wpq 2024-09-01T02:04:09.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-01T02:04:09.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:04:09.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:04:09.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c9bf559-8a3d-468a-8127-619ffd12a10f 2024-09-01T02:04:09.417 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 7c9bf559-8a3d-468a-8127-619ffd12a10f 2024-09-01T02:04:09.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7c9bf559-8a3d-468a-8127-619ffd12a10f' 2024-09-01T02:04:09.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:04:09.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZy9NmCCrKGRAA74e7jtddxOB+VxNuBY6y+g== 2024-09-01T02:04:09.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZy9NmCCrKGRAA74e7jtddxOB+VxNuBY6y+g=="}' 2024-09-01T02:04:09.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c9bf559-8a3d-468a-8127-619ffd12a10f -i td/ceph-helpers/1/new.json 2024-09-01T02:04:09.998 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:04:10.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-01T02:04:10.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCZy9NmCCrKGRAA74e7jtddxOB+VxNuBY6y+g== --osd-uuid 7c9bf559-8a3d-468a-8127-619ffd12a10f 2024-09-01T02:04:10.039 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:10.038+0000 7f0ec1c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:10.041 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:10.040+0000 7f0ec1c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:10.042 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:10.041+0000 7f0ec1c87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:10.042 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:10.042+0000 7f0ec1c87640 -1 bdev(0x55ff61ee9c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:04:10.043 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:10.042+0000 7f0ec1c87640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-01T02:04:15.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-01T02:04:15.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:04:15.960 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T02:04:15.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:04:15.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:04:16.523 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T02:04:16.524 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:04:16.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-01T02:04:16.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:04:16.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:04:16.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:04:16.551 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:16.550+0000 7f30454ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:16.553 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:16.552+0000 7f30454ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:16.554 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:16.553+0000 7f30454ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:16.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:04:16.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:04:16.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:04:16.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:16.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:17.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:17.896 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:17.896+0000 7f30454ef640 -1 Falling back to public interface 2024-09-01T02:04:18.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:18.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:18.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:04:18.394 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:04:18.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:18.394 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:18.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:19.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:19.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:19.837 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:04:19.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:04:19.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:19.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:20.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:20.799 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:20.798+0000 7f30454ef640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:04:21.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:21.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:21.252 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:04:21.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:04:21.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:21.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:21.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:22.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:22.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:22.671 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:04:22.671 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:04:22.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:22.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:23.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:24.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:24.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:24.105 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:04:24.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:04:24.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:24.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:24.736 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:25.737 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:25.737 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:04:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:04:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:25.738 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:26.189 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:27.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:27.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:27.191 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:04:27.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:04:27.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:27.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:27.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:28.636 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:28.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:28.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:04:28.637 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:04:28.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:28.637 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:29.049 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:30.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:30.051 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:30.051 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:04:30.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:04:30.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:30.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:30.472 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:31.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:31.473 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:31.473 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:04:31.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:04:31.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:31.474 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:32.045 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:33.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:33.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:33.047 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:04:33.047 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:04:33.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:33.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:33.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:34.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:34.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:34.646 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:04:34.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:04:34.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:34.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:35.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:36.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:36.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:36.830 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:04:36.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:04:36.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:36.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:37.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:38.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:38.779 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:38.780 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:04:38.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:04:38.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:38.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:39.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:40.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:40.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:40.665 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:04:40.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:04:40.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:40.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:41.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:42.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:42.560 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:42.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:04:42.561 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:04:42.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:42.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:43.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:44.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:44.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:44.101 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:04:44.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:04:44.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:44.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:04:44.600 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3521173063,v1:127.0.0.1:6811/3521173063] [v2:127.0.0.1:6812/3521173063,v1:127.0.0.1:6813/3521173063] exists,up 7c9bf559-8a3d-468a-8127-619ffd12a10f 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: test_get_op_scheduler: get_op_scheduler 1 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: get_op_scheduler: local id=1 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2024-09-01T02:04:44.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-osd.1.asok 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:04:44.603 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-osd.1.asok config get osd_op_queue 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:04:44.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:04:44.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:04:44.796 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:04:44.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:04:44.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:04:44.797 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:04:44.797 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:04:44.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:04:44.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:04:44.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:04:44.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:04:44.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:04:44.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:04:44.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:04:44.806 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:04:44.806 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:44.806 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:44.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:04:44.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:04:44.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:04:44.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:04:44.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:04:44.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:04:44.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:04:44.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:04:44.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:04:44.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:04:44.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:04:44.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:04:44.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:04:44.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:04:44.813 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:04:44.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:04:44.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:04:44.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:04:44.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:04:44.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:44.816 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:44.816 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:04:44.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:04:44.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:04:44.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:04:44.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:04:44.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:44.818 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:44.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:04:44.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:45.289 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:45.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:04:45.523 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:04:45.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:04:45.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:04:45.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:04:45.524 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:45.525 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:04:45.526 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:04:45.526 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:04:45.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:04:45.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:04:45.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:46.555 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:46.556 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:04:46.556 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:04:46.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-01T02:04:46.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:04:46.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:04:46.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:04:46.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:04:46.588 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:04:46.589 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=708d521a-b017-4f28-a641-58e0d00b3c0d 2024-09-01T02:04:46.590 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 708d521a-b017-4f28-a641-58e0d00b3c0d 2024-09-01T02:04:46.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 708d521a-b017-4f28-a641-58e0d00b3c0d' 2024-09-01T02:04:46.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:04:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+y9Nmmm0lJBAAdPpiQIcoiKapLf90+eGG4g== 2024-09-01T02:04:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+y9Nmmm0lJBAAdPpiQIcoiKapLf90+eGG4g=="}' 2024-09-01T02:04:46.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 708d521a-b017-4f28-a641-58e0d00b3c0d -i td/ceph-helpers/0/new.json 2024-09-01T02:04:46.881 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:04:46.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:04:46.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+y9Nmmm0lJBAAdPpiQIcoiKapLf90+eGG4g== --osd-uuid 708d521a-b017-4f28-a641-58e0d00b3c0d 2024-09-01T02:04:46.922 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:46.921+0000 7f271a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:46.958 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:46.957+0000 7f271a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:46.959 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:46.958+0000 7f271a4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:46.959 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:46.958+0000 7f271a4f1640 -1 bdev(0x55d26bcd7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:04:46.959 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:46.958+0000 7f271a4f1640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:04:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:04:52.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:04:52.464 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:04:52.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:04:52.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:04:53.075 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:04:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:04:53.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:04:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:04:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:04:53.076 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:04:53.101 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:53.101+0000 7f3d9d487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:53.103 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:53.103+0000 7f3d9d487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:53.105 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:53.104+0000 7f3d9d487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:04:53.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:53.510 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:04:53.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:04:53.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:53.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:53.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:54.447 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:54.446+0000 7f3d9d487640 -1 Falling back to public interface 2024-09-01T02:04:54.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:54.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:54.945 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:04:54.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:04:54.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:54.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:55.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:56.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:56.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:56.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:04:56.388 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:04:56.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:56.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:56.846 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:57.449 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:04:57.448+0000 7f3d9d487640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:04:57.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:57.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:57.847 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:04:57.848 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:04:57.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:57.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:58.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:04:59.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:04:59.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:04:59.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:04:59.546 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:04:59.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:04:59.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:04:59.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:00.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:01.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:02.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:02.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:02.489 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:05:02.490 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:05:02.490 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:02.490 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:02.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:03.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:03.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:03.933 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:05:03.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:05:03.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:03.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:04.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:05.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:05.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:05.346 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:05:05.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:05:05.346 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:05.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:05.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:06.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:06.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:06.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:05:06.762 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:05:06.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:06.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:07.312 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:08.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:08.313 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:08.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:05:08.314 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:05:08.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:08.314 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:08.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:09.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:09.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:09.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:05:09.732 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:05:09.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:09.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:10.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:11.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:11.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:11.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:05:11.322 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:05:11.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:11.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:12.013 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:13.014 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:13.014 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:13.015 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:05:13.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:05:13.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:13.016 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:13.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:14.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:14.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:14.727 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:05:14.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:05:14.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:14.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:15.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:16.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:16.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:16.178 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:05:16.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:05:16.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:16.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:16.840 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:17.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:17.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:17.842 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:05:17.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:05:17.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:17.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:18.716 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:19.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:19.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:19.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:05:19.718 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:05:19.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:19.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:20.197 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2461977917,v1:127.0.0.1:6803/2461977917] [v2:127.0.0.1:6804/2461977917,v1:127.0.0.1:6805/2461977917] exists,up 708d521a-b017-4f28-a641-58e0d00b3c0d 2024-09-01T02:05:20.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:05:20.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:05:20.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:05:20.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-01T02:05:20.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:05:20.665 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:05:20.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:05:20.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:05:21.239 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:05:21.254 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:05:22.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:05:23.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-01T02:05:23.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:05:23.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:05:23.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:05:23.340 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:05:23.341 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:05:23.341 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:05:23.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:05:23.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:05:23.341 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:05:23.435 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:05:23.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:05:23.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:05:23.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:05:23.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:05:23.436 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:05:23.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:05:23.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:05:23.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:05:23.865 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-01T02:05:23.982 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:05:23.983 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:05:23.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-01T02:05:23.983 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:05:23.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-01T02:05:23.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-01T02:05:23.985 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836484 2024-09-01T02:05:23.985 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:05:24.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-01T02:05:24.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:05:25.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:05:25.384 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:05:25.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-01T02:05:25.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:05:25.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:05:25.820 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:05:26.368 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:05:26.794 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:05:26.794 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:05:26.794 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:05:26.794 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:05:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:05:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:05:27.350 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:05:27.350 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-01T02:05:27.350 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2024-09-01T02:05:27.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-01T02:05:27.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-01T02:05:27.351 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2024-09-01T02:05:27.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:05:27.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:05:27.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:05:27.875 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:05:27.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:05:27.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:05:27.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:05:27.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:05:27.877 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:05:27.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:05:27.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:05:27.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:05:27.879 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:05:27.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:05:27.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:05:27.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:05:27.885 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:05:27.886 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:27.886 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:27.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_osds td/ceph-helpers 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2024-09-01T02:05:27.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:05:27.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:05:27.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:05:27.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:05:27.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:05:27.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:05:27.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:05:27.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:05:27.891 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:05:27.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:05:27.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:05:27.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:05:27.892 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:05:27.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:05:27.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:05:27.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:05:27.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:05:27.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:27.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:27.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:05:27.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:05:27.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:05:27.896 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:05:27.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:05:27.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:27.897 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:27.897 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:05:27.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:05:27.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:05:27.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:05:27.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-01T02:05:28.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:05:28.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:05:28.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:05:28.363 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:05:28.364 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:28.364 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:28.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:05:28.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-01T02:05:28.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:05:28.547 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:05:28.548 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:05:28.549 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:28.549 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:28.549 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:05:28.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:05:28.549 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:05:28.630 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:05:28.631 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:05:28.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2024-09-01T02:05:28.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:05:28.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:05:28.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:05:28.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:05:28.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:05:28.710 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:29.574 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:05:29.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:05:29.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:05:29.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2024-09-01T02:05:29.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:05:29.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:05:29.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:05:29.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:05:29.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=133c6bc9-cf1a-4715-9a9d-a4d0394073d8 2024-09-01T02:05:29.612 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 133c6bc9-cf1a-4715-9a9d-a4d0394073d8' 2024-09-01T02:05:29.613 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 133c6bc9-cf1a-4715-9a9d-a4d0394073d8 2024-09-01T02:05:29.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:05:29.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpy9Nmv8RsJRAAJmv2RlD9Eo5siXvzb1gMiA== 2024-09-01T02:05:29.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpy9Nmv8RsJRAAJmv2RlD9Eo5siXvzb1gMiA=="}' 2024-09-01T02:05:29.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 133c6bc9-cf1a-4715-9a9d-a4d0394073d8 -i td/ceph-helpers/0/new.json 2024-09-01T02:05:29.906 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:05:29.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:05:29.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpy9Nmv8RsJRAAJmv2RlD9Eo5siXvzb1gMiA== --osd-uuid 133c6bc9-cf1a-4715-9a9d-a4d0394073d8 2024-09-01T02:05:29.946 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:29.945+0000 7f6f2b61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:29.973 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:29.972+0000 7f6f2b61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:29.974 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:29.973+0000 7f6f2b61c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:29.974 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:29.973+0000 7f6f2b61c640 -1 bdev(0x559cefb79c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:05:29.975 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:29.974+0000 7f6f2b61c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:05:35.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:05:35.938 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:05:35.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:05:35.939 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:05:35.940 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:05:36.521 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:05:36.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:05:36.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:05:36.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:05:36.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:05:36.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:05:36.549 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:36.548+0000 7f8f40411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:36.561 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:36.560+0000 7f8f40411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:36.562 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:36.561+0000 7f8f40411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:36.955 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:05:36.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:05:36.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:36.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:37.391 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:38.155 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:38.154+0000 7f8f40411640 -1 Falling back to public interface 2024-09-01T02:05:38.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:38.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:38.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:05:38.393 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:05:38.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:38.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:38.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:39.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:39.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:39.836 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:05:39.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:05:39.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:39.837 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:40.285 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:40.897 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:05:40.896+0000 7f8f40411640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:05:41.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:41.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:41.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:05:41.287 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:05:41.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:41.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:41.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:42.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:42.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:42.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:05:42.960 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:05:42.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:42.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:43.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:44.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:44.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:44.417 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:05:44.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:05:44.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:44.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:44.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:45.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:45.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:45.936 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:05:45.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:05:45.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:45.937 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:46.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:47.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:47.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:47.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:05:47.368 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:05:47.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:47.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:47.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:48.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:48.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:48.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:05:48.802 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:05:48.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:48.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:49.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:50.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:50.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:50.241 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:05:50.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:05:50.241 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:50.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:50.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:51.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:51.768 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:51.768 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:05:51.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:05:51.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:51.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:52.246 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:53.247 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:53.248 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:53.248 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:05:53.249 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:05:53.249 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:53.249 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:53.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:54.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:54.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:54.980 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:05:54.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:05:54.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:54.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:55.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:56.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:56.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:56.677 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:05:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:05:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:56.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:57.270 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:05:58.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:05:58.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:05:58.272 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:05:58.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:05:58.272 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:05:58.273 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:05:59.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:00.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:00.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:00.375 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:06:00.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:06:00.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:00.376 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:06:01.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:02.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:02.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:02.369 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:06:02.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:06:02.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:02.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:06:02.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:03.872 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:03.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:03.873 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:06:03.873 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:06:03.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:03.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:06:04.284 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2331826096,v1:127.0.0.1:6803/2331826096] [v2:127.0.0.1:6804/2331826096,v1:127.0.0.1:6805/2331826096] exists,up 133c6bc9-cf1a-4715-9a9d-a4d0394073d8 2024-09-01T02:06:04.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:06:04.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:06:04.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:06:04.290 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2024-09-01T02:06:04.291 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:06:04.292 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:06:04.293 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=600860b1-ac80-4fd9-91d2-9c520d349583 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 600860b1-ac80-4fd9-91d2-9c520d349583' 2024-09-01T02:06:04.294 INFO:tasks.workunit.client.0.smithi037.stdout:add osd1 600860b1-ac80-4fd9-91d2-9c520d349583 2024-09-01T02:06:04.295 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:06:04.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMzNNms+9GEhAAfxeDUdX0391LyjwV0dzOPw== 2024-09-01T02:06:04.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMzNNms+9GEhAAfxeDUdX0391LyjwV0dzOPw=="}' 2024-09-01T02:06:04.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 600860b1-ac80-4fd9-91d2-9c520d349583 -i td/ceph-helpers/1/new.json 2024-09-01T02:06:04.773 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:06:04.784 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2024-09-01T02:06:04.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMzNNms+9GEhAAfxeDUdX0391LyjwV0dzOPw== --osd-uuid 600860b1-ac80-4fd9-91d2-9c520d349583 2024-09-01T02:06:04.814 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:04.813+0000 7f818f589640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:04.818 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:04.817+0000 7f818f589640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:04.819 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:04.818+0000 7f818f589640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:04.819 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:04.818+0000 7f818f589640 -1 bdev(0x5633035ffc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:06:04.819 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:04.819+0000 7f818f589640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-01T02:06:10.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-01T02:06:10.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:06:10.787 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd1 key to auth repository 2024-09-01T02:06:10.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:06:10.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:06:11.355 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.1 2024-09-01T02:06:11.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:06:11.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:06:11.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:06:11.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:06:11.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:06:11.382 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:11.381+0000 7fe48e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:11.388 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:11.387+0000 7fe48e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:11.389 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:11.388+0000 7fe48e4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:11.834 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:06:11.835 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:06:11.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:11.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:12.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:12.733 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:12.733+0000 7fe48e4e2640 -1 Falling back to public interface 2024-09-01T02:06:13.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:13.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:13.322 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:06:13.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:06:13.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:13.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:13.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:14.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:14.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:14.788 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:06:14.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:06:14.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:14.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:15.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:15.969 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:15.969+0000 7fe48e4e2640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:06:16.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:16.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:16.233 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:06:16.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:06:16.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:16.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:16.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:17.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:17.777 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:17.777 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:06:17.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:06:17.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:17.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:18.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:19.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:19.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:19.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:06:19.194 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:06:19.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:19.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:19.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:20.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:20.610 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:20.610 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:06:20.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:06:20.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:20.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:21.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:22.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:22.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:22.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:06:22.114 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:06:22.114 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:22.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:22.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:23.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:23.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:23.679 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:06:23.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:06:23.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:23.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:24.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:25.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:25.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:25.104 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:06:25.105 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:06:25.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:25.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:25.525 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:26.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:26.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:26.526 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:06:26.527 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:06:26.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:26.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:26.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:27.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:27.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:27.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:06:27.944 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:06:27.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:27.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:28.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:29.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:29.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:29.403 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:06:29.403 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:06:29.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:29.404 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:29.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:30.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:30.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:30.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:06:30.936 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:06:30.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:30.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:31.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:32.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:32.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:32.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:06:32.699 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:06:32.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:32.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:33.632 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:34.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:34.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:34.634 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:06:34.634 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:06:34.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:34.635 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:35.364 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:36.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:36.366 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:36.367 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:06:36.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:06:36.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:36.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:37.160 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:38.161 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:06:38.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:06:38.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:38.162 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:38.660 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:06:39.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:06:39.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:06:39.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:06:39.662 INFO:tasks.workunit.client.0.smithi037.stdout:18 2024-09-01T02:06:39.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:06:39.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:06:40.116 INFO:tasks.workunit.client.0.smithi037.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1050902909,v1:127.0.0.1:6811/1050902909] [v2:127.0.0.1:6812/1050902909,v1:127.0.0.1:6813/1050902909] exists,up 600860b1-ac80-4fd9-91d2-9c520d349583 2024-09-01T02:06:40.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:06:40.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:06:40.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:06:40.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2024-09-01T02:06:40.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:06:40.510 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:06:40.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:06:40.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:06:41.037 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:06:41.048 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:06:42.050 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:06:43.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2024-09-01T02:06:43.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:06:43.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:06:43.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:06:43.676 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:06:43.677 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:06:43.677 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:06:43.677 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:06:43.677 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:06:43.677 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:06:43.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:06:43.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:06:43.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:06:43.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:06:43.772 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:06:43.772 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:06:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:06:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:1' 2024-09-01T02:06:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:06:44.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:06:44.253 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:06:44.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-01T02:06:44.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-01T02:06:44.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-01T02:06:44.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:06:44.369 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672967' 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-01T02:06:44.486 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:06:44.487 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:06:44.487 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-01T02:06:44.487 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:06:44.488 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-01T02:06:44.488 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836498 2024-09-01T02:06:44.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-01T02:06:44.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:06:44.921 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-01T02:06:44.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:06:45.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:06:45.923 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:06:46.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-01T02:06:46.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:06:46.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-01T02:06:46.372 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:06:46.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T02:06:46.373 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-01T02:06:46.374 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:06:46.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-01T02:06:46.375 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.1 seq 42949672967 2024-09-01T02:06:46.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-01T02:06:46.375 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:06:46.776 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-01T02:06:46.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:06:46.776 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:06:46.777 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:06:47.324 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:06:47.325 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:06:47.757 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:06:47.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:06:47.757 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:06:47.758 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:06:48.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:06:48.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:06:48.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:06:48.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2024-09-01T02:06:48.317 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:06:48.830 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:06:48.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:06:48.841 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:06:49.356 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:06:49.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:06:50.368 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:06:51.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2024-09-01T02:06:51.601 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2024-09-01T02:06:51.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2024-09-01T02:06:51.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-01T02:06:51.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2024-09-01T02:06:51.602 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-01T02:06:52.062 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:0' 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:06:52.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:06:52.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:06:52.169 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:06:52.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:06:52.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:06:52.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:06:52.171 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:06:52.171 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:06:52.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:06:52.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:06:52.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:06:52.173 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:06:52.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:06:52.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:06:52.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:06:52.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:06:52.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.184 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_pg td/ceph-helpers 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:06:52.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:06:52.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:06:52.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:06:52.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:06:52.188 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:06:52.189 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:06:52.189 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:06:52.189 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:06:52.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:06:52.190 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:06:52.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:06:52.191 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:06:52.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:06:52.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:06:52.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:06:52.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:06:52.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.193 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.193 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:06:52.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:06:52.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:06:52.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:06:52.195 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:06:52.196 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.196 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:06:52.197 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:06:52.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:06:52.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:06:52.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:06:52.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:06:52.674 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:06:52.674 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.674 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:52.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:06:52.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:06:52.866 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:06:52.867 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:06:52.867 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.867 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.868 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:06:52.868 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:06:52.868 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:06:52.952 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:06:52.953 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:06:53.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:06:53.899 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:06:53.899 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:53.900 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:06:53.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:06:53.932 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:06:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:06:53.934 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:06:53.935 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d47e61d1-57f3-4125-9a8c-e8cb5d5429d7 2024-09-01T02:06:53.936 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 d47e61d1-57f3-4125-9a8c-e8cb5d5429d7 2024-09-01T02:06:53.936 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d47e61d1-57f3-4125-9a8c-e8cb5d5429d7' 2024-09-01T02:06:53.936 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:06:53.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9zNNmWtq7OBAAT0g4UGaBrzjRb605G2nU6g== 2024-09-01T02:06:53.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9zNNmWtq7OBAAT0g4UGaBrzjRb605G2nU6g=="}' 2024-09-01T02:06:53.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d47e61d1-57f3-4125-9a8c-e8cb5d5429d7 -i td/ceph-helpers/0/new.json 2024-09-01T02:06:54.240 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:06:54.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:06:54.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9zNNmWtq7OBAAT0g4UGaBrzjRb605G2nU6g== --osd-uuid d47e61d1-57f3-4125-9a8c-e8cb5d5429d7 2024-09-01T02:06:54.280 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:54.278+0000 7fe5297a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:54.300 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:54.298+0000 7fe5297a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:54.301 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:54.299+0000 7fe5297a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:06:54.302 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:54.300+0000 7fe5297a2640 -1 bdev(0x5590bfe59c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:06:54.302 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:06:54.300+0000 7fe5297a2640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:06:59.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:06:59.829 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:06:59.830 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:06:59.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:06:59.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:07:00.443 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:07:00.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:07:00.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:07:00.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:07:00.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:07:00.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:07:00.470 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:00.469+0000 7f668e154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:00.479 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:00.478+0000 7f668e154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:00.480 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:00.479+0000 7f668e154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:00.927 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:07:00.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:07:00.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:00.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:01.352 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:02.076 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:02.076+0000 7f668e154640 -1 Falling back to public interface 2024-09-01T02:07:02.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:02.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:02.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:07:02.354 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:07:02.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:02.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:02.786 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:03.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:03.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:03.788 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:07:03.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:07:03.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:03.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:04.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:04.666 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:04.665+0000 7f668e154640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:07:05.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:05.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:05.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:07:05.213 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:07:05.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:05.213 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:05.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:06.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:06.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:06.787 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:07:06.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:07:06.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:06.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:07.215 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:08.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:08.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:08.217 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:07:08.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:07:08.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:08.218 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:08.748 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:09.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:09.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:09.750 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:07:09.750 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:07:09.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:09.751 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:10.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:11.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:11.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:11.174 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:07:11.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:07:11.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:11.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:11.592 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:12.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:12.593 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:12.594 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:07:12.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:07:12.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:12.594 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:13.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:14.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:14.018 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:14.019 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:07:14.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:07:14.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:14.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:14.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:15.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:15.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:15.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:07:15.577 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:07:15.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:15.577 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:16.001 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:17.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:17.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:17.003 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:07:17.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:07:17.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:17.004 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:17.516 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:18.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:18.518 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:18.518 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:07:18.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:07:18.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:18.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:19.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:20.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:20.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:20.281 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:07:20.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:07:20.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:20.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:21.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:22.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:22.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:22.074 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:07:22.075 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:07:22.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:22.075 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:22.510 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:23.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:23.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:23.512 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:07:23.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:07:23.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:23.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:24.778 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:25.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:25.780 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:25.780 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:07:25.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:07:25.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:25.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:26.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:27.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:27.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:27.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:07:27.953 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:07:27.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:27.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:28.417 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/168749261,v1:127.0.0.1:6803/168749261] [v2:127.0.0.1:6804/168749261,v1:127.0.0.1:6805/168749261] exists,up d47e61d1-57f3-4125-9a8c-e8cb5d5429d7 2024-09-01T02:07:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:07:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:07:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:07:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2024-09-01T02:07:28.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:07:28.840 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:07:28.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:07:28.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:07:29.455 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:07:29.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:07:30.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:07:31.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2024-09-01T02:07:31.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:07:31.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:07:31.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:07:31.639 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:07:31.640 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:07:31.640 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:07:31.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:07:31.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:07:31.640 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:07:31.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:07:31.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:07:31.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:07:31.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:07:31.735 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:07:31.735 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:07:32.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:07:32.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:07:32.147 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:07:32.147 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:07:32.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836491 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836491 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836491' 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836491 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:07:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:07:32.265 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836491 2024-09-01T02:07:32.265 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:07:32.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836491 2024-09-01T02:07:32.266 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-01T02:07:32.266 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836491 2024-09-01T02:07:32.266 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:07:32.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-01T02:07:32.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:07:33.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:07:33.701 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:07:34.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-01T02:07:34.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:07:35.167 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-01T02:07:35.167 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:07:35.611 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-01T02:07:35.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:07:35.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:07:35.611 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:07:36.163 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:07:36.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:07:36.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:07:36.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:07:36.554 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2024-09-01T02:07:37.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-01T02:07:37.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2024-09-01T02:07:37.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:07:37.557 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:07:37.558 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:07:37.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:07:37.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:07:37.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:07:37.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:07:37.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:07:37.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:07:37.665 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:07:37.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:07:37.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:07:37.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:07:37.667 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:07:37.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:07:37.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:07:37.668 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:07:37.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:07:37.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:37.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:37.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:07:37.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_primary td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:07:37.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:07:37.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:07:37.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:07:37.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:07:37.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:07:37.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:07:37.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:07:37.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:07:37.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:07:37.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:07:37.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:07:37.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:07:37.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:07:37.682 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:07:37.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:07:37.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:07:37.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:07:37.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:37.684 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:37.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:07:37.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:07:37.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:07:37.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:07:37.686 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:07:37.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:37.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:37.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:07:37.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:07:37.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:07:37.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:07:37.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:07:37.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:07:38.160 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:07:38.160 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:07:38.160 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:07:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:07:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:38.161 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:07:38.161 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:07:38.361 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:07:38.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:07:38.363 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:07:38.364 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:07:38.365 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:38.365 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:38.365 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:07:38.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:07:38.365 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:07:38.452 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:07:38.453 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:38.453 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:38.453 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:07:38.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:07:38.453 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:07:38.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2024-09-01T02:07:38.533 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:07:38.534 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:39.385 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:39.386 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:07:39.386 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:07:39.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:07:39.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2024-09-01T02:07:39.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:07:39.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:07:39.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:07:39.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:07:39.418 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:07:39.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:07:39.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:07:39.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:07:39.420 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:07:39.420 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab056566-bdf2-4a85-a062-1daddbf2c765 2024-09-01T02:07:39.420 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 ab056566-bdf2-4a85-a062-1daddbf2c765 2024-09-01T02:07:39.421 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ab056566-bdf2-4a85-a062-1daddbf2c765' 2024-09-01T02:07:39.421 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:07:39.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBrzNNmpML+GRAA9HYgnPk/Is+qCn5u/8KH3A== 2024-09-01T02:07:39.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBrzNNmpML+GRAA9HYgnPk/Is+qCn5u/8KH3A=="}' 2024-09-01T02:07:39.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab056566-bdf2-4a85-a062-1daddbf2c765 -i td/ceph-helpers/0/new.json 2024-09-01T02:07:39.709 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:07:39.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:07:39.720 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrzNNmpML+GRAA9HYgnPk/Is+qCn5u/8KH3A== --osd-uuid ab056566-bdf2-4a85-a062-1daddbf2c765 2024-09-01T02:07:39.750 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:39.748+0000 7f3b4a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:39.778 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:39.777+0000 7f3b4a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:39.779 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:39.778+0000 7f3b4a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:39.780 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:39.779+0000 7f3b4a20d640 -1 bdev(0x560d637efc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:07:39.780 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:39.779+0000 7f3b4a20d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:07:45.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:07:45.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:07:45.281 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:07:45.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:07:45.282 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:07:45.848 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:07:45.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:07:45.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:07:45.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:07:45.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:07:45.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:07:45.876 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:45.875+0000 7f3358c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:45.883 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:45.882+0000 7f3358c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:45.884 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:45.883+0000 7f3358c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:07:46.286 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:07:46.287 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:07:46.288 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:46.288 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:46.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:46.980 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:46.979+0000 7f3358c18640 -1 Falling back to public interface 2024-09-01T02:07:47.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:47.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:47.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:07:47.731 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:07:47.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:47.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:48.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:49.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:49.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:49.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:07:49.182 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:07:49.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:49.182 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:49.602 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:49.716 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:07:49.715+0000 7f3358c18640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:07:50.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:50.603 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:50.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:07:50.604 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:07:50.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:50.604 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:51.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:52.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:52.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:52.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:07:52.038 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:07:52.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:52.039 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:52.460 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:53.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:53.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:53.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:07:53.462 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:07:53.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:53.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:53.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:54.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:54.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:54.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:07:54.907 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:07:54.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:54.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:55.320 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:56.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:56.321 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:56.321 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:07:56.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:07:56.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:56.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:56.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:57.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:57.805 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:57.806 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:07:57.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:07:57.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:57.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:58.234 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:07:59.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:07:59.236 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:07:59.236 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:07:59.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:07:59.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:07:59.237 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:07:59.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:00.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:00.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:00.808 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:08:00.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:08:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:00.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:01.223 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:02.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:03.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:04.200 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:04.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:04.201 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:08:04.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:08:04.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:04.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:05.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:06.198 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:06.198 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:08:06.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:06.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:08:06.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:06.199 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:07.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:08.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:08.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:08.105 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:08:08.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:08:08.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:08.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:09.303 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:10.304 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:10.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:10.305 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:08:10.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:08:10.305 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:10.306 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:11.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:12.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:12.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:12.099 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:08:12.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:08:12.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:12.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:12.581 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:13.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:13.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:13.582 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:08:13.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:08:13.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:13.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:14.028 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3512015602,v1:127.0.0.1:6803/3512015602] [v2:127.0.0.1:6804/3512015602,v1:127.0.0.1:6805/3512015602] exists,up ab056566-bdf2-4a85-a062-1daddbf2c765 2024-09-01T02:08:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:08:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:08:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:08:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2024-09-01T02:08:14.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:08:14.526 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:08:14.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:08:14.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:08:15.031 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:08:15.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:08:16.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:08:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2024-09-01T02:08:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:08:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:08:17.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:08:17.107 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:08:17.108 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:17.108 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:17.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:17.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:17.108 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:08:17.203 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:08:17.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:08:17.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:08:17.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:08:17.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-01T02:08:17.731 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:08:17.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:08:17.732 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-01T02:08:17.733 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:08:17.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-01T02:08:17.734 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836487 2024-09-01T02:08:17.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-01T02:08:17.734 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:08:18.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-01T02:08:18.184 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:08:19.185 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:08:19.186 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:08:19.609 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-01T02:08:19.609 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:08:19.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:08:19.610 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:08:20.166 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:08:20.167 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:08:20.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:08:20.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:08:20.575 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:08:20.576 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:08:21.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2024-09-01T02:08:21.127 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:08:21.582 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:08:21.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:08:21.583 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:08:21.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:08:21.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:08:21.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:08:21.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:08:21.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:08:21.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:08:21.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:08:21.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:08:21.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:08:21.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:08:21.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:08:21.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:08:21.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:08:21.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:08:21.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:08:21.699 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:21.699 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:21.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:08:21.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: test_get_timeout_delays: get_timeout_delays 1 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.701 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.713 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: test_get_timeout_delays: get_timeout_delays 5 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.714 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-01T02:08:21.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: test_get_timeout_delays: get_timeout_delays 6 2024-09-01T02:08:21.730 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.730 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.730 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.749 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: test_get_timeout_delays: get_timeout_delays 7 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.750 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-01T02:08:21.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: test_get_timeout_delays: get_timeout_delays 8 2024-09-01T02:08:21.774 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.774 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.774 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.775 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.798 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-01T02:08:21.798 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-01T02:08:21.799 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.799 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.824 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.825 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-01T02:08:21.855 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1614: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-01T02:08:21.855 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.856 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1614: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-01T02:08:21.893 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-01T02:08:21.894 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.894 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-01T02:08:21.931 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-01T02:08:21.932 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.932 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.932 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.933 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:21.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-01T02:08:21.971 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-01T02:08:21.972 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:21.972 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:21.972 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:21.972 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:21.972 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:22.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-01T02:08:22.022 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-01T02:08:22.022 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:22.023 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:22.023 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:22.023 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:22.023 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:22.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-01T02:08:22.101 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-01T02:08:22.101 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:08:22.102 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:08:22.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:08:22.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:08:22.102 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_is_clean td/ceph-helpers 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:08:22.335 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:08:22.336 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:08:22.337 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:08:22.338 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:08:22.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:08:22.338 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:08:22.339 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:08:22.340 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:08:22.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:08:22.341 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:08:22.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:08:22.343 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:08:22.343 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:22.344 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:22.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:08:22.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:08:22.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:08:22.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:08:22.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:08:22.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:08:22.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:08:22.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:08:22.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:08:22.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:08:22.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:08:23.002 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:08:23.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:08:23.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:08:23.003 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:23.004 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:08:23.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:08:23.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:08:23.086 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:23.087 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:23.087 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:08:23.087 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:08:23.087 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:08:23.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2024-09-01T02:08:23.165 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:08:23.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:08:23.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:08:23.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:08:23.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:08:23.166 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:08:24.034 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:08:24.035 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:08:24.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:08:24.067 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:08:24.068 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:08:24.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:08:24.070 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:08:24.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ce83764-3adc-486d-9483-47f801e8e553 2024-09-01T02:08:24.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7ce83764-3adc-486d-9483-47f801e8e553' 2024-09-01T02:08:24.070 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 7ce83764-3adc-486d-9483-47f801e8e553 2024-09-01T02:08:24.071 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:08:24.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYzNNmRgwnBRAAG9/NSbWhMqiHuNnawYtYfw== 2024-09-01T02:08:24.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYzNNmRgwnBRAAG9/NSbWhMqiHuNnawYtYfw=="}' 2024-09-01T02:08:24.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ce83764-3adc-486d-9483-47f801e8e553 -i td/ceph-helpers/0/new.json 2024-09-01T02:08:24.426 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:08:24.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:08:24.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYzNNmRgwnBRAAG9/NSbWhMqiHuNnawYtYfw== --osd-uuid 7ce83764-3adc-486d-9483-47f801e8e553 2024-09-01T02:08:24.467 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:24.466+0000 7fcb35afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:24.495 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:24.494+0000 7fcb35afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:24.497 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:24.496+0000 7fcb35afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:24.497 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:24.496+0000 7fcb35afe640 -1 bdev(0x561b3605bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:08:24.498 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:24.496+0000 7fcb35afe640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:08:29.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:08:29.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:08:29.857 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:08:29.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:08:29.857 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:08:30.437 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:08:30.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:08:30.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:08:30.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:08:30.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:08:30.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:08:30.465 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:30.464+0000 7f3075e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:30.466 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:30.466+0000 7f3075e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:30.467 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:30.467+0000 7f3075e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:08:30.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:08:30.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:08:30.848 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:08:30.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:08:30.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:08:30.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:30.849 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:08:30.849 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:08:30.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:30.850 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:31.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:31.308 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:31.307+0000 7f3075e0d640 -1 Falling back to public interface 2024-09-01T02:08:32.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:32.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:32.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:08:32.263 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:08:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:32.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:32.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:33.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:33.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:33.707 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:08:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:08:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:33.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:34.119 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:34.367 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:08:34.366+0000 7f3075e0d640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:08:35.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:35.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:35.121 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:08:35.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:08:35.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:35.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:35.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:36.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:36.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:36.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:08:36.546 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:08:36.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:36.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:36.973 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:37.976 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:37.976 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:37.976 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:08:37.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:08:37.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:37.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:38.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:39.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:39.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:39.508 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:08:39.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:08:39.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:39.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:39.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:40.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:40.930 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:40.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:08:40.931 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:08:40.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:40.931 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:41.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:42.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:42.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:42.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:08:42.374 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:08:42.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:42.374 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:42.790 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:43.791 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:43.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:43.792 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:08:43.792 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:08:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:44.330 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:45.332 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:45.332 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:45.332 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:08:45.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:08:45.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:45.333 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:45.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:46.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:46.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:46.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:08:46.905 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:08:46.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:46.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:47.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:48.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:48.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:48.568 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:08:48.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:08:48.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:48.569 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:49.132 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:50.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:50.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:50.133 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:08:50.133 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:08:50.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:50.134 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:50.744 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:51.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:51.745 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:51.746 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:08:51.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:08:51.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:51.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:52.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:53.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:53.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:53.451 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:08:53.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:08:53.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:53.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:54.194 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:55.195 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:55.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:55.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:08:55.196 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:08:55.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:55.196 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:55.804 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:08:56.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:08:56.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:08:56.806 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:08:56.806 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:08:56.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:08:56.807 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:08:57.228 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3822629620,v1:127.0.0.1:6803/3822629620] [v2:127.0.0.1:6804/3822629620,v1:127.0.0.1:6805/3822629620] exists,up 7ce83764-3adc-486d-9483-47f801e8e553 2024-09-01T02:08:57.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:08:57.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:08:57.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:08:57.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2024-09-01T02:08:57.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:08:57.688 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:08:57.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:08:57.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:08:58.203 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:08:58.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:08:59.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:09:00.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2024-09-01T02:09:00.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:09:00.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:09:00.418 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:09:00.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:09:00.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:09:00.419 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:09:00.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:09:00.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:09:00.420 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:09:00.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:09:00.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:09:00.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:09:00.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:09:00.514 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:09:00.515 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:09:00.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:09:00.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:09:00.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:09:00.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:09:01.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-01T02:09:01.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-01T02:09:01.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-01T02:09:01.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:09:01.081 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-01T02:09:01.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:09:01.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:09:01.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-01T02:09:01.082 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:09:01.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-01T02:09:01.083 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836484 2024-09-01T02:09:01.084 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-01T02:09:01.084 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:09:01.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 2 -lt 21474836484 2024-09-01T02:09:01.495 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:09:02.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:09:02.497 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:09:02.965 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-01T02:09:02.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:09:02.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:09:02.965 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:09:03.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:09:03.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:09:03.643 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:09:03.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:09:03.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:09:03.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:09:03.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:09:03.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:09:04.052 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:09:04.052 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:09:04.053 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:09:04.053 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:09:04.624 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:09:05.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2024-09-01T02:09:05.177 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2024-09-01T02:09:05.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2024-09-01T02:09:05.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:09:05.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:09:05.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:09:05.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:09:05.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:09:05.599 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:09:05.600 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:09:05.600 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:09:05.600 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:09:05.600 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:09:05.704 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:09:05.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:09:05.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:09:05.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:09:05.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:09:05.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:09:05.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:09:05.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:05.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:09:05.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:09:05.708 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:05.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:09:05.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:09:05.709 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:09:05.715 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:09:05.715 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:05.715 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:05.716 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:09:05.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:09:05.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:09:05.717 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_kill_daemon td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:09:05.718 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:09:05.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:09:05.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:09:05.719 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:09:05.719 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:09:05.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:09:05.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:09:05.721 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:09:05.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:05.721 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:09:05.722 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:09:05.722 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:05.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:09:05.723 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:09:05.724 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:09:05.725 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:09:05.725 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:05.725 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:05.725 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:09:05.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:09:05.727 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:09:05.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:09:05.728 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:09:05.728 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:05.728 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:05.728 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:09:05.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:09:05.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:09:05.729 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:09:05.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:06.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:09:06.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:06.225 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:06.225 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:06.225 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:06.226 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:06.226 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:06.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:06.408 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:09:06.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:09:06.408 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:09:06.409 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:09:06.409 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:09:06.409 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:09:06.410 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:09:06.492 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:09:06.493 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:09:06.572 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:07.449 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:07.450 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:09:07.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:09:07.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-01T02:09:07.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:09:07.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:09:07.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:09:07.481 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:09:07.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85424f83-797e-49e8-b41e-452f64202bb3 2024-09-01T02:09:07.483 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 85424f83-797e-49e8-b41e-452f64202bb3' 2024-09-01T02:09:07.484 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 85424f83-797e-49e8-b41e-452f64202bb3 2024-09-01T02:09:07.484 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:09:07.500 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDDzNNmGg3GHRAAHXVqBBNcp/5DhHNznAMvZw== 2024-09-01T02:09:07.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDDzNNmGg3GHRAAHXVqBBNcp/5DhHNznAMvZw=="}' 2024-09-01T02:09:07.501 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85424f83-797e-49e8-b41e-452f64202bb3 -i td/ceph-helpers/0/new.json 2024-09-01T02:09:07.799 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:09:07.809 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:09:07.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDzNNmGg3GHRAAHXVqBBNcp/5DhHNznAMvZw== --osd-uuid 85424f83-797e-49e8-b41e-452f64202bb3 2024-09-01T02:09:07.840 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:07.838+0000 7fc2bd4e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:07.868 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:07.866+0000 7fc2bd4e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:07.869 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:07.867+0000 7fc2bd4e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:07.869 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:07.867+0000 7fc2bd4e6640 -1 bdev(0x55e1d0f69c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:09:07.869 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:07.868+0000 7fc2bd4e6640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:09:13.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:09:13.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:09:13.646 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:09:13.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:09:13.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:09:14.215 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:09:14.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:09:14.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:09:14.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:09:14.216 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:09:14.217 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:09:14.243 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:14.241+0000 7f8d7922f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:14.254 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:14.252+0000 7f8d7922f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:14.255 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:14.253+0000 7f8d7922f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:14.642 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:14.643 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:09:14.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:09:14.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:14.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:15.068 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:15.614 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:15.614+0000 7f8d7922f640 -1 Falling back to public interface 2024-09-01T02:09:16.069 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:16.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:16.070 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:09:16.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:09:16.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:16.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:16.511 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:17.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:17.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:17.512 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:09:17.513 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:09:17.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:17.513 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:17.961 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:18.574 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:18.573+0000 7f8d7922f640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:09:18.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:18.963 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:18.963 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:09:18.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:09:18.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:18.964 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:19.375 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:20.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:20.377 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:20.377 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:09:20.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:09:20.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:20.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:20.813 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:21.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:21.814 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:21.815 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:09:21.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:09:21.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:21.815 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:22.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:23.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:23.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:23.389 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:09:23.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:09:23.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:23.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:23.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:24.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:24.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:24.843 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:09:24.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:09:24.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:24.844 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:25.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:26.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:26.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:26.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:09:26.389 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:09:26.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:26.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:26.800 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:27.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:27.801 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:27.801 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:09:27.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:09:27.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:27.802 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:28.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:29.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:29.325 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:29.325 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:09:29.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:09:29.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:29.326 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:29.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:30.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:30.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:30.742 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:09:30.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:09:30.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:30.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:31.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:32.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:32.388 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:32.389 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:09:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:09:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:33.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:34.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:34.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:34.499 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:09:34.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:09:34.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:34.499 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:35.135 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:36.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:36.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:36.136 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:09:36.137 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:09:36.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:36.137 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:36.785 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:37.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:37.787 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:37.787 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:09:37.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:09:37.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:37.788 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:38.732 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:39.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:39.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:39.733 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:09:39.734 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:09:39.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:39.734 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:40.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:41.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:41.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:41.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:09:41.820 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:09:41.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:41.821 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:42.274 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1881729884,v1:127.0.0.1:6803/1881729884] [v2:127.0.0.1:6804/1881729884,v1:127.0.0.1:6805/1881729884] exists,up 85424f83-797e-49e8-b41e-452f64202bb3 2024-09-01T02:09:42.274 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:09:42.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:09:42.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:09:42.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2024-09-01T02:09:42.275 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2024-09-01T02:09:42.275 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-01T02:09:42.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-01T02:09:42.276 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-01T02:09:42.276 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-01T02:09:42.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=112660 2024-09-01T02:09:42.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2024-09-01T02:09:42.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2024-09-01T02:09:42.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-01T02:09:42.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:42.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 112660 2024-09-01T02:09:42.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:42.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:42.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2024-09-01T02:09:43.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 112660 2024-09-01T02:09:43.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2024-09-01T02:09:43.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2024-09-01T02:09:43.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-01T02:09:43.279 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=112660 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 112660 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.281 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-01T02:09:43.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 112660 2024-09-01T02:09:43.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.382 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 112660 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2024-09-01T02:09:43.584 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-01T02:09:43.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-01T02:09:43.585 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-01T02:09:43.585 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-01T02:09:43.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=111710 2024-09-01T02:09:43.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-01T02:09:43.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-01T02:09:43.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-01T02:09:43.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 111710 2024-09-01T02:09:43.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.588 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 111710 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2024-09-01T02:09:43.689 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-01T02:09:43.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-01T02:09:43.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-01T02:09:43.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=111625 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 111625 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.692 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2024-09-01T02:09:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 111625 2024-09-01T02:09:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2024-09-01T02:09:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2024-09-01T02:09:43.793 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2024-09-01T02:09:43.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2024-09-01T02:09:43.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 111625 2024-09-01T02:09:43.994 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2024-09-01T02:09:43.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2024-09-01T02:09:43.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2024-09-01T02:09:43.995 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:09:48.999 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:09:49.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:09:49.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:09:49.000 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:09:49.003 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:09:49.004 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:09:49.005 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:09:49.005 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:09:49.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:09:49.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:09:49.006 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:09:49.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:49.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:09:49.008 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:09:49.008 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:49.009 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:09:49.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:09:49.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:09:49.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:09:49.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.017 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.017 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_kill_daemons td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:09:49.019 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:09:49.020 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:09:49.021 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:09:49.022 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:09:49.023 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:09:49.023 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:09:49.023 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:09:49.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:49.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:09:49.024 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:09:49.025 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:09:49.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:09:49.030 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:09:49.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.031 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.031 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:09:49.033 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.496 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.497 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:49.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:09:49.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:09:49.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:09:49.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:09:49.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:09:49.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:09:49.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:09:49.699 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:09:49.700 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:09:49.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:09:49.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:09:49.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:09:49.788 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:09:49.789 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:09:49.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:50.729 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:09:50.730 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:09:50.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-01T02:09:50.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:09:50.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:09:50.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2c418e9-939d-416f-a0c3-ec76fdca0999 2024-09-01T02:09:50.764 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 f2c418e9-939d-416f-a0c3-ec76fdca0999 2024-09-01T02:09:50.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f2c418e9-939d-416f-a0c3-ec76fdca0999' 2024-09-01T02:09:50.765 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:09:50.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDuzNNmkdB3LhAAhEf0iN0FUYlui+M9wplGbg== 2024-09-01T02:09:50.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDuzNNmkdB3LhAAhEf0iN0FUYlui+M9wplGbg=="}' 2024-09-01T02:09:50.781 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2c418e9-939d-416f-a0c3-ec76fdca0999 -i td/ceph-helpers/0/new.json 2024-09-01T02:09:51.072 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:09:51.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:09:51.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuzNNmkdB3LhAAhEf0iN0FUYlui+M9wplGbg== --osd-uuid f2c418e9-939d-416f-a0c3-ec76fdca0999 2024-09-01T02:09:51.112 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:51.111+0000 7fe71bc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:51.140 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:51.139+0000 7fe71bc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:51.141 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:51.140+0000 7fe71bc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:51.142 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:51.141+0000 7fe71bc12640 -1 bdev(0x55fe55831c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:09:51.142 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:51.141+0000 7fe71bc12640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:09:56.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:09:56.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:09:56.944 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:09:56.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:09:56.944 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:09:57.508 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:09:57.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:09:57.508 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:09:57.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:09:57.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:09:57.509 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:09:57.535 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:57.534+0000 7f3cec41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:57.537 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:57.536+0000 7f3cec41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:57.538 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:57.537+0000 7f3cec41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:09:57.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:09:57.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:09:57.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:09:57.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:09:57.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:09:57.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:57.943 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:09:57.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:09:57.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:57.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:58.397 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:09:58.396+0000 7f3cec41c640 -1 Falling back to public interface 2024-09-01T02:09:58.409 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:09:59.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:09:59.410 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:09:59.411 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:09:59.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:09:59.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:09:59.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:09:59.851 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:00.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:00.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:00.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:10:00.853 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:10:00.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:00.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:01.265 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:01.723 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:01.722+0000 7f3cec41c640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:10:02.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:02.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:02.267 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:10:02.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:10:02.267 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:02.268 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:02.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:03.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:03.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:03.765 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:10:03.765 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:10:03.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:03.766 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:04.186 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:05.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:05.187 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:05.188 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:10:05.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:10:05.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:05.188 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:05.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:06.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:06.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:06.687 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:10:06.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:10:06.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:06.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:07.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:08.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:08.109 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:08.110 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:10:08.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:10:08.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:08.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:08.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:09.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:09.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:09.607 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:10:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:10:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:09.608 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:10.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:11.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:11.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:11.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:10:11.030 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:10:11.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:11.030 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:11.543 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:12.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:12.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:12.544 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:10:12.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:10:12.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:12.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:12.958 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:13.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:13.959 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:13.960 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:10:13.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:10:13.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:13.960 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:14.584 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:15.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:15.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:15.586 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:10:15.586 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:10:15.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:15.587 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:16.261 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:17.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:17.262 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:17.263 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:10:17.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:10:17.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:17.263 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:18.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:19.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:19.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:19.228 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:10:19.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:10:19.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:19.229 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:20.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:21.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:21.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:21.043 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:10:21.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:10:21.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:21.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:21.773 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:22.774 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:22.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:22.775 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:10:22.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:10:22.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:22.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:23.898 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:24.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:24.900 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:24.901 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:10:24.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:10:24.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:24.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:25.347 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/193673281,v1:127.0.0.1:6803/193673281] [v2:127.0.0.1:6804/193673281,v1:127.0.0.1:6805/193673281] exists,up f2c418e9-939d-416f-a0c3-ec76fdca0999 2024-09-01T02:10:25.348 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:25.349 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:26.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2024-09-01T02:10:26.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-01T02:10:26.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:26.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:26.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:26.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:26.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:26.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:26.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:26.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:10:26.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-01T02:10:26.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:26.769 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:26.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:26.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:26.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:26.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:10:26.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:31.881 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:31.885 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:10:31.886 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:10:31.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:10:31.888 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:10:31.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:10:31.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:10:31.889 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:10:31.889 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:10:31.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:10:31.890 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:10:31.890 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:10:31.891 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:10:31.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:10:31.892 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:10:31.898 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:10:31.899 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:31.899 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:31.899 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:10:31.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:10:31.901 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:10:31.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:10:31.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:10:31.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:10:31.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:10:31.903 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:10:31.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:10:31.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:10:31.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:10:31.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:10:31.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:10:31.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:10:31.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:10:31.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:10:31.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:10:31.906 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:10:31.907 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:10:31.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:10:31.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:10:31.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:10:31.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:31.910 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:31.910 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:10:31.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:10:31.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:10:31.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:10:31.912 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:10:31.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:31.913 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:31.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:10:31.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:10:32.388 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:10:32.388 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:10:32.388 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:10:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:10:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:32.389 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:10:32.389 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:10:32.580 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:10:32.581 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:10:32.581 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:10:32.581 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:10:32.581 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:10:32.582 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:10:32.582 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:10:32.583 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:10:32.672 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:32.673 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:32.673 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:10:32.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:10:32.673 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:10:32.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:10:32.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:10:32.755 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:10:33.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:10:33.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:10:33.613 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:10:33.613 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:10:33.613 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:33.614 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:33.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:10:33.614 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:10:33.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:10:33.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-01T02:10:33.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:10:33.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:10:33.647 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:10:33.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=82a37b03-8494-4c29-9f25-4856714e5b24 2024-09-01T02:10:33.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 82a37b03-8494-4c29-9f25-4856714e5b24' 2024-09-01T02:10:33.650 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 82a37b03-8494-4c29-9f25-4856714e5b24 2024-09-01T02:10:33.650 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:10:33.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZzdNm3u+fJxAAU8YwDzzk47+d7WyobqgevA== 2024-09-01T02:10:33.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZzdNm3u+fJxAAU8YwDzzk47+d7WyobqgevA=="}' 2024-09-01T02:10:33.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 82a37b03-8494-4c29-9f25-4856714e5b24 -i td/ceph-helpers/0/new.json 2024-09-01T02:10:33.996 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:10:34.005 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:10:34.006 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZzdNm3u+fJxAAU8YwDzzk47+d7WyobqgevA== --osd-uuid 82a37b03-8494-4c29-9f25-4856714e5b24 2024-09-01T02:10:34.036 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:34.035+0000 7ffa1120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:34.065 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:34.064+0000 7ffa1120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:34.066 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:34.066+0000 7ffa1120f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:34.067 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:34.066+0000 7ffa1120f640 -1 bdev(0x560f79df7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:10:34.067 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:34.066+0000 7ffa1120f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:10:40.010 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:10:40.011 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:10:40.011 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:10:40.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:10:40.012 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:10:40.574 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:10:40.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:10:40.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:10:40.575 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:10:40.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:10:40.576 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:10:40.602 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:40.601+0000 7f751a813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:40.611 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:40.610+0000 7f751a813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:40.612 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:40.612+0000 7f751a813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:10:41.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:10:41.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:10:41.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:10:41.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:41.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:41.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:41.940 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:41.939+0000 7f751a813640 -1 Falling back to public interface 2024-09-01T02:10:42.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:42.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:42.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:10:42.468 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:10:42.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:42.469 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:42.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:43.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:43.911 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:43.911 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:10:43.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:10:43.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:43.912 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:44.322 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:44.430 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:10:44.429+0000 7f751a813640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:10:45.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:45.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:45.323 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:10:45.324 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:10:45.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:45.324 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:45.740 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:46.741 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:46.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:46.742 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:10:46.742 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:10:46.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:46.743 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:47.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:48.231 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:10:48.232 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:10:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:48.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:48.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:49.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:49.647 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:10:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:10:49.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:49.648 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:50.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:51.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:51.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:51.207 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:10:51.207 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:10:51.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:51.208 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:51.623 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:52.624 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:52.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:52.625 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:10:52.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:10:52.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:52.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:53.173 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:54.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:54.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:54.174 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:10:54.175 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:10:54.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:54.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:54.613 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:55.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:55.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:55.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:10:55.615 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:10:55.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:55.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:56.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:57.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:57.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:57.109 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:10:57.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:10:57.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:57.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:57.664 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:10:58.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:10:58.665 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:10:58.666 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:10:58.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:10:58.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:10:58.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:10:59.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:00.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:00.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:00.416 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:11:00.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:11:00.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:00.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:01.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:02.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:02.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:02.252 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:11:02.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:11:02.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:02.253 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:02.986 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:03.988 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:03.988 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:11:03.988 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:03.988 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:11:03.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:03.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:04.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:05.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:05.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:05.504 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:11:05.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:11:05.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:05.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:06.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:07.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:07.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:07.042 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:11:07.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:11:07.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:07.043 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:07.495 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2253426414,v1:127.0.0.1:6803/2253426414] [v2:127.0.0.1:6804/2253426414,v1:127.0.0.1:6805/2253426414] exists,up 82a37b03-8494-4c29-9f25-4856714e5b24 2024-09-01T02:11:07.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:11:07.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:11:07.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:11:07.496 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2024-09-01T02:11:07.497 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:11:07.902 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:11:07.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:11:07.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:11:08.425 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:11:08.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:11:09.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:11:10.565 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2024-09-01T02:11:10.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:11:10.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:11:10.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:11:10.566 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:11:10.566 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:11:10.567 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:11:10.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:11:10.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:11:10.567 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:11:10.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:11:10.661 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:11:10.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:11:10.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:11:10.662 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:11:10.662 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:11:11.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:11:11.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:11:11.093 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:11:11.093 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:11:11.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-01T02:11:11.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-01T02:11:11.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-01T02:11:11.211 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:11:11.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-01T02:11:11.212 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:11:11.212 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:11:11.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-01T02:11:11.213 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:11:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-01T02:11:11.214 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-01T02:11:11.214 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836485 2024-09-01T02:11:11.214 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:11:11.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-01T02:11:11.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:11:12.616 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:11:12.616 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:11:13.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836485 2024-09-01T02:11:13.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:11:13.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:11:13.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:11:13.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:11:13.685 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:11:14.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:11:14.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:11:14.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:11:14.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:11:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:11:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:11:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:11:14.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-01T02:11:14.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-01T02:11:14.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2024-09-01T02:11:14.817 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2024-09-01T02:11:14.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-01T02:11:14.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-01T02:11:14.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-01T02:11:14.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-01T02:11:14.818 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:11:14.819 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:11:14.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:11:14.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:11:14.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:11:15.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:11:15.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-01T02:11:15.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-01T02:11:15.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-01T02:11:15.760 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T02:11:15.761 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:11:15.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:start osd.0 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:11:15.763 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-01T02:11:15.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T02:11:15.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T02:11:15.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T02:11:15.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T02:11:15.790 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:15.788+0000 7fced1b2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:15.791 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:15.791+0000 7fced1b2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:15.793 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:15.792+0000 7fced1b2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:16.205 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:0 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:16.206 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:16.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:16.881 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:16.880+0000 7fced1b2e640 -1 Falling back to public interface 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:1 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:17.649 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:18.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:2 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:19.125 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:19.540 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:19.743 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:19.742+0000 7fced1b2e640 -1 osd.0 13 log_to_monitors true 2024-09-01T02:11:20.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:20.541 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:20.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:11:20.542 INFO:tasks.workunit.client.0.smithi037.stderr:3 2024-09-01T02:11:20.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:20.542 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:21.028 INFO:tasks.workunit.client.0.smithi037.stderr:osd.0 up in weight 1 up_from 16 up_thru 16 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/496823042,v1:127.0.0.1:6803/496823042] [v2:127.0.0.1:6804/496823042,v1:127.0.0.1:6805/496823042] exists,up 82a37b03-8494-4c29-9f25-4856714e5b24 2024-09-01T02:11:21.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:11:21.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:11:21.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:11:21.028 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:11:21.029 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:11:21.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:11:21.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:11:21.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:11:21.563 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:11:21.563 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:11:21.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476739 2024-09-01T02:11:21.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476739 2024-09-01T02:11:21.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-68719476739' 2024-09-01T02:11:21.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:11:21.678 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-68719476739 2024-09-01T02:11:21.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:11:21.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:11:21.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-68719476739 2024-09-01T02:11:21.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:11:21.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476739 2024-09-01T02:11:21.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-09-01T02:11:21.681 INFO:tasks.workunit.client.0.smithi037.stderr:waiting osd.0 seq 68719476739 2024-09-01T02:11:21.681 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:11:22.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-01T02:11:22.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:11:22.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:11:22.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:11:22.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:11:22.663 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:11:22.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:11:22.663 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:11:22.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:11:22.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:11:22.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:11:22.664 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:11:23.091 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:11:23.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:11:23.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:11:23.091 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:11:23.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:11:23.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:11:23.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:11:23.643 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:11:23.644 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:11:23.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:11:23.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:11:23.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:11:23.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:11:23.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-01T02:11:23.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-01T02:11:23.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T02:11:23.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-01T02:11:23.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T02:11:23.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-01T02:11:23.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-01T02:11:24.304 INFO:tasks.workunit.client.0.smithi037.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-01T02:11:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-01T02:11:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2024-09-01T02:11:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2024-09-01T02:11:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2024-09-01T02:11:24.570 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:11:24.571 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:11:24.678 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:11:24.678 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:11:24.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:11:24.679 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:11:24.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:11:24.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:11:24.680 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:11:24.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:11:24.681 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:11:24.682 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:11:24.682 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:11:24.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:11:24.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:11:24.684 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:11:24.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:11:24.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:24.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:24.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-09-01T02:11:24.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: setup td/ceph-helpers 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:11:24.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:11:24.695 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:11:24.696 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:11:24.696 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:11:24.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:11:24.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:11:24.698 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:11:24.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:11:24.698 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:11:24.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:11:24.699 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:11:24.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:11:24.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:11:24.701 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:11:24.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:11:24.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:24.702 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:24.702 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:11:24.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:11:24.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:11:24.703 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:11:24.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:11:24.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:24.705 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:24.705 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:11:24.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:11:24.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:11:24.708 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:25.171 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:25.172 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:25.172 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:11:25.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:11:25.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:11:25.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:11:25.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:11:25.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:11:25.356 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:11:25.360 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:11:25.360 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:11:25.361 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:11:25.445 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:25.446 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:11:25.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:11:25.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:11:25.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-09-01T02:11:25.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:11:25.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:11:25.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:11:25.527 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:11:25.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:11:25.528 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:26.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:26.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:11:26.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:11:26.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-09-01T02:11:26.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:11:26.412 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:11:26.413 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:11:26.414 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=febbb313-0ac8-4fe4-b260-31a7440ce4c7 2024-09-01T02:11:26.415 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 febbb313-0ac8-4fe4-b260-31a7440ce4c7 2024-09-01T02:11:26.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 febbb313-0ac8-4fe4-b260-31a7440ce4c7' 2024-09-01T02:11:26.415 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:11:26.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBOzdNmPvaoGRAA0KyVypAWiBCBDWGHfjJM1A== 2024-09-01T02:11:26.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBOzdNmPvaoGRAA0KyVypAWiBCBDWGHfjJM1A=="}' 2024-09-01T02:11:26.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new febbb313-0ac8-4fe4-b260-31a7440ce4c7 -i td/ceph-helpers/0/new.json 2024-09-01T02:11:26.696 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:11:26.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:11:26.707 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOzdNmPvaoGRAA0KyVypAWiBCBDWGHfjJM1A== --osd-uuid febbb313-0ac8-4fe4-b260-31a7440ce4c7 2024-09-01T02:11:26.736 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:26.735+0000 7f8de801c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:26.756 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:26.755+0000 7f8de801c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:26.757 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:26.756+0000 7f8de801c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:26.757 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:26.757+0000 7f8de801c640 -1 bdev(0x564be7561c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:11:26.758 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:26.757+0000 7f8de801c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:11:32.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:11:32.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:11:32.521 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:11:32.521 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:11:32.522 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:11:33.088 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:11:33.089 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:11:33.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:11:33.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:11:33.089 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:11:33.090 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:11:33.117 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:33.115+0000 7f26c420e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:33.127 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:33.125+0000 7f26c420e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:33.128 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:33.126+0000 7f26c420e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:11:33.519 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:33.520 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:11:33.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:11:33.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:33.520 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:33.922 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:34.477 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:34.475+0000 7f26c420e640 -1 Falling back to public interface 2024-09-01T02:11:34.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:34.923 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:34.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:11:34.924 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:11:34.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:34.924 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:35.367 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:36.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:36.369 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:36.369 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:11:36.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:11:36.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:36.370 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:36.810 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:37.455 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:11:37.454+0000 7f26c420e640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:11:37.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:37.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:37.811 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:11:37.812 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:11:37.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:37.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:38.225 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:39.226 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:39.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:39.227 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:11:39.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:11:39.227 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:39.228 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:39.677 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:40.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:40.679 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:40.679 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:11:40.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:11:40.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:40.680 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:41.230 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:42.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:42.232 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:42.232 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:11:42.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:11:42.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:42.233 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:42.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:43.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:43.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:43.646 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:11:43.647 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:11:43.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:43.647 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:44.063 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:45.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:45.064 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:45.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:11:45.065 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:11:45.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:45.065 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:45.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:46.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:46.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:46.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:11:46.545 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:11:46.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:46.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:46.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:47.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:47.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:47.970 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:11:47.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:11:47.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:47.970 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:48.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:49.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:49.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:49.440 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:11:49.440 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:11:49.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:49.441 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:49.976 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:50.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:50.977 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:50.977 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:11:50.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:11:50.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:50.978 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:51.638 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:52.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:52.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:52.639 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:11:52.639 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:11:52.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:52.640 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:53.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:54.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:54.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:54.952 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:11:54.952 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:11:54.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:54.953 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:55.422 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:56.423 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:56.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:56.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:11:56.424 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:11:56.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:56.424 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:57.179 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:11:58.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:11:58.180 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:11:58.181 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:11:58.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:11:58.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:11:58.181 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:11:59.275 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:00.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:00.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:00.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:12:00.277 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:12:00.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:00.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:00.698 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1559595803,v1:127.0.0.1:6803/1559595803] [v2:127.0.0.1:6804/1559595803,v1:127.0.0.1:6805/1559595803] exists,up febbb313-0ac8-4fe4-b260-31a7440ce4c7 2024-09-01T02:12:00.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:12:00.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:12:00.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:12:00.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: test_pg_schedule_scrub: create_rbd_pool 2024-09-01T02:12:00.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:12:01.145 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:12:01.156 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:12:01.157 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:12:01.655 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:12:01.666 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:12:02.667 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:12:03.893 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_pg_schedule_scrub: wait_for_clean 2024-09-01T02:12:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:12:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:12:03.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:12:03.894 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:12:03.895 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:12:03.895 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:12:03.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:12:03.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:12:03.895 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:12:03.990 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:12:04.428 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:12:04.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:12:04.429 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:12:04.429 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:12:04.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-01T02:12:04.547 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-01T02:12:04.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-01T02:12:04.548 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:12:04.548 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-01T02:12:04.548 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:12:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:12:04.550 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-01T02:12:04.550 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:12:04.550 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-01T02:12:04.551 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-01T02:12:04.551 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836485 2024-09-01T02:12:04.552 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:12:04.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-01T02:12:04.971 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:12:05.972 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:12:05.973 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:12:06.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-01T02:12:06.415 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:12:07.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-01T02:12:07.416 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:12:07.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-01T02:12:07.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:12:07.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:12:07.867 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:12:08.419 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:12:08.843 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:12:08.843 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:12:08.843 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:12:08.843 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:12:09.392 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_schedule_scrub: local pgid=1.0 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:09.393 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:09.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_schedule_scrub: local last_scrub=2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:09.812 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_schedule_scrub: ceph pg scrub 1.0 2024-09-01T02:12:10.197 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_schedule_scrub: wait_for_scrub 1.0 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:10.209 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:10.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:10.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:10.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:10.210 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:10.605 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:01.485527+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:10.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:11.606 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:11.607 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:11.607 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:11.607 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:11.608 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:11.608 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:11.608 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:12.013 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:01.485527+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:12.013 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:13.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:13.015 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:13.015 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:13.015 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:13.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:13.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:13.016 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:13.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:01.485527+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:13.432 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:14.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:14.433 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:14.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:14.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:14.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:14.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:14.434 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:14.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:01.485527+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:14.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:15.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:15.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:15.877 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:15.877 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:15.877 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:15.878 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:15.878 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:16.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:01.485527+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:16.283 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:17.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:17.284 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:12:17.285 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:17.285 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:17.285 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:17.285 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:17.286 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:11.059338+0000 '>' 2024-09-01T02:12:01.485527+0000 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:12:17.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: TIMEOUT=1 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: pg_scrub 1.0 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:17.799 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:18.191 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-01T02:12:11.059338+0000 2024-09-01T02:12:18.192 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-01T02:12:18.579 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-01T02:12:18.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-01T02:12:11.059338+0000 2024-09-01T02:12:18.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:12:18.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:12:11.059338+0000 2024-09-01T02:12:18.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:12:18.591 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:12:18.987 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:11.059338+0000 '>' 2024-09-01T02:12:11.059338+0000 2024-09-01T02:12:18.988 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:12:19.989 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:12:19.990 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:12:20.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:12:20.097 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:12:20.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:12:20.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:12:20.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:12:20.099 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:12:20.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:12:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:12:20.100 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:12:20.101 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:12:20.101 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:12:20.102 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:12:20.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:12:20.103 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:12:20.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:12:20.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.110 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.110 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_pg_scrub td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_pg_scrub: local dir=td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: setup td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:12:20.112 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:12:20.113 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:12:20.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:12:20.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:12:20.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:12:20.113 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:12:20.114 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:12:20.115 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:12:20.115 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:12:20.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:12:20.116 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:12:20.116 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:12:20.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:12:20.117 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:12:20.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:12:20.118 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:12:20.118 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:12:20.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:12:20.120 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:12:20.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:12:20.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.121 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:12:20.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:12:20.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:12:20.122 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:12:20.123 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:12:20.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.124 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:12:20.126 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:12:20.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:12:20.590 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:12:20.808 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:12:20.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:12:20.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:12:20.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:12:20.809 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.810 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.811 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:12:20.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:12:20.811 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:12:20.901 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:12:20.902 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:12:20.903 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:12:20.903 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:20.903 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:20.904 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:12:20.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:12:20.904 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:12:20.984 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:12:20.985 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:12:21.841 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:12:21.842 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:12:21.874 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1902: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:12:21.875 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:12:21.876 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:12:21.877 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:12:21.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77a77367-abe8-4f4e-9fcd-04e7f8cced41 2024-09-01T02:12:21.878 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 77a77367-abe8-4f4e-9fcd-04e7f8cced41 2024-09-01T02:12:21.878 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 77a77367-abe8-4f4e-9fcd-04e7f8cced41' 2024-09-01T02:12:21.879 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:12:21.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFzdNmRZRBNRAAPBBDxTnXe7Q/7IRRBqtW+g== 2024-09-01T02:12:21.894 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFzdNmRZRBNRAAPBBDxTnXe7Q/7IRRBqtW+g=="}' 2024-09-01T02:12:21.895 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77a77367-abe8-4f4e-9fcd-04e7f8cced41 -i td/ceph-helpers/0/new.json 2024-09-01T02:12:22.165 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:12:22.175 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:12:22.176 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFzdNmRZRBNRAAPBBDxTnXe7Q/7IRRBqtW+g== --osd-uuid 77a77367-abe8-4f4e-9fcd-04e7f8cced41 2024-09-01T02:12:22.205 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:22.204+0000 7f3e2e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:22.233 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:22.232+0000 7f3e2e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:22.234 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:22.234+0000 7f3e2e21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:22.235 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:22.234+0000 7f3e2e21c640 -1 bdev(0x563a64735c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:12:22.235 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:22.234+0000 7f3e2e21c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:12:27.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:12:27.928 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:12:27.929 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:12:27.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:12:27.929 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:12:28.502 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:12:28.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:12:28.503 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:12:28.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:12:28.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:12:28.504 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:12:28.529 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:28.528+0000 7f4677e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:28.546 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:28.545+0000 7f4677e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:28.547 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:28.546+0000 7f4677e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:12:28.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:12:28.968 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:28.969 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:29.385 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:30.149 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:30.148+0000 7f4677e2c640 -1 Falling back to public interface 2024-09-01T02:12:30.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:30.386 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:30.387 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:12:30.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:12:30.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:30.387 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:30.828 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:31.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:31.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:31.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:12:31.830 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:12:31.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:31.831 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:32.250 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:32.741 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:12:32.740+0000 7f4677e2c640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:12:33.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:33.251 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:33.252 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:12:33.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:12:33.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:33.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:33.864 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:34.865 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:34.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:34.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:12:34.866 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:12:34.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:34.866 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:35.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:36.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:36.309 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:36.309 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:12:36.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:12:36.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:36.310 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:36.913 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:37.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:37.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:37.915 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:12:37.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:12:37.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:37.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:38.353 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:39.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:39.354 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:39.355 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:12:39.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:12:39.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:39.355 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:39.906 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:40.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:40.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:40.908 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:12:40.908 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:12:40.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:40.909 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:41.356 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:42.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:42.358 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:42.358 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:12:42.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:12:42.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:42.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:42.852 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:43.853 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:43.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:43.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:12:43.854 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:12:43.854 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:43.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:44.277 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:45.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:45.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:45.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:12:45.279 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:12:45.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:45.279 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:46.105 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:47.106 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:47.107 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:12:47.107 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:12:47.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:47.108 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:47.939 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:48.940 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:48.940 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:48.941 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:12:48.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:12:48.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:48.941 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:49.625 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:50.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:50.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:50.627 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:12:50.627 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:12:50.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:50.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:51.566 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:52.567 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:52.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:52.568 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:12:52.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:12:52.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:52.568 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:53.552 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:54.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:54.553 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:54.554 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:12:54.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:12:54.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:54.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:55.371 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:12:56.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:12:56.372 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:12:56.372 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:12:56.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:12:56.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:12:56.373 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:12:56.829 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1756731445,v1:127.0.0.1:6803/1756731445] [v2:127.0.0.1:6804/1756731445,v1:127.0.0.1:6805/1756731445] exists,up 77a77367-abe8-4f4e-9fcd-04e7f8cced41 2024-09-01T02:12:56.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:12:56.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:12:56.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:12:56.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: test_pg_scrub: create_rbd_pool 2024-09-01T02:12:56.830 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:12:57.231 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:12:57.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:12:57.242 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:12:57.741 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:12:57.752 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:12:58.754 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:12:59.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_pg_scrub: wait_for_clean 2024-09-01T02:12:59.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:12:59.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:12:59.783 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:12:59.784 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:12:59.784 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:12:59.784 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:12:59.784 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:12:59.784 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:12:59.785 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:12:59.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:12:59.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:12:59.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:12:59.879 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:12:59.880 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:12:59.880 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:13:00.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:13:00.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:13:00.319 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:13:00.319 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:13:00.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-01T02:13:00.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-01T02:13:00.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-01T02:13:00.436 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:13:00.437 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-01T02:13:00.437 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:13:00.437 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:13:00.437 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-01T02:13:00.438 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:13:00.438 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-01T02:13:00.439 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-01T02:13:00.439 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836486 2024-09-01T02:13:00.439 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:13:00.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-01T02:13:00.887 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:13:01.888 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:13:01.888 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:13:02.347 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-01T02:13:02.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:13:02.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:13:02.347 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:13:02.904 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:13:02.905 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:13:03.307 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:13:03.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:13:03.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:13:03.307 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:13:03.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:13:03.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:13:03.855 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: pg_scrub 1.0 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:03.856 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:04.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:04.252 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-01T02:13:04.633 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:04.645 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:04.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:04.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:04.646 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:05.040 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:57.575201+0000 '>' 2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:05.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:06.041 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:06.042 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:06.042 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:06.043 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:06.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:57.575201+0000 '>' 2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:06.465 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:07.466 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:07.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:07.467 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:07.467 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:07.468 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:07.468 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:07.468 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:07.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:12:57.575201+0000 '>' 2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:07.870 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:08.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:08.871 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:08.872 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:08.872 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:08.872 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:08.873 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:08.873 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:04.799628+0000 '>' 2024-09-01T02:12:57.575201+0000 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:13:09.278 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: TIMEOUT=1 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: pg_scrub 1.0 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:09.383 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:09.384 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:09.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-01T02:13:04.799628+0000 2024-09-01T02:13:09.775 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-01T02:13:10.158 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-01T02:13:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-01T02:13:04.799628+0000 2024-09-01T02:13:10.169 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:13:04.799628+0000 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:10.170 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:10.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:04.799628+0000 '>' 2024-09-01T02:13:04.799628+0000 2024-09-01T02:13:10.559 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:11.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:11.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:13:11.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-01T02:13:11.561 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: teardown td/ceph-helpers 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:13:11.562 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:13:11.669 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:13:11.669 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:13:11.670 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:13:11.670 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:13:11.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:13:11.672 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:13:11.672 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:13:11.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:13:11.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:13:11.673 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:13:11.674 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:13:11.674 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:13:11.675 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:13:11.676 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:13:11.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:13:11.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:11.683 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:11.683 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:13:11.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:13:11.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:13:11.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: for func in $funcs 2024-09-01T02:13:11.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: test_repair td/ceph-helpers 2024-09-01T02:13:11.685 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_repair: local dir=td/ceph-helpers 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: setup td/ceph-helpers 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:13:11.686 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:13:11.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:13:11.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:13:11.687 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:13:11.687 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:13:11.688 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:13:11.688 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:13:11.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:13:11.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:13:11.690 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:13:11.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:13:11.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:13:11.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:13:11.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:13:11.691 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:13:11.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:13:11.693 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:13:11.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:13:11.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:11.694 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:11.694 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:13:11.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:13:11.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:13:11.695 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2024-09-01T02:13:11.697 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:13:11.697 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:11.697 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:11.697 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.66036 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:13:11.699 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2024-09-01T02:13:11.700 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:13:12.177 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:13:12.178 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-01T02:13:12.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:13:12.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:13:12.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:13:12.379 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:13:12.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:13:12.380 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:13:12.381 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:13:12.382 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get fsid 2024-09-01T02:13:12.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:13:12.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:13:12.473 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.66036/ceph-mon.a.asok 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:13:12.474 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.66036/ceph-mon.a.asok config get mon_host 2024-09-01T02:13:12.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: test_repair: run_mgr td/ceph-helpers x 2024-09-01T02:13:12.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2024-09-01T02:13:12.554 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:13:12.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:13:12.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:13:12.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2024-09-01T02:13:12.555 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:13:13.410 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:13:13.410 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:13:13.410 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:13:13.411 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: test_repair: run_osd td/ceph-helpers 0 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-01T02:13:13.443 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:13:13.444 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:13:13.445 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:13:13.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2024-09-01T02:13:13.446 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:13:13.446 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c834b82-b111-4177-9652-f9b811e9f1b3 2024-09-01T02:13:13.446 INFO:tasks.workunit.client.0.smithi037.stdout:add osd0 0c834b82-b111-4177-9652-f9b811e9f1b3 2024-09-01T02:13:13.447 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0c834b82-b111-4177-9652-f9b811e9f1b3' 2024-09-01T02:13:13.447 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:13:13.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC5zdNmArSBGxAAZ6ZX95MUJ6nBbsAF3nycvw== 2024-09-01T02:13:13.462 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC5zdNmArSBGxAAZ6ZX95MUJ6nBbsAF3nycvw=="}' 2024-09-01T02:13:13.463 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c834b82-b111-4177-9652-f9b811e9f1b3 -i td/ceph-helpers/0/new.json 2024-09-01T02:13:13.736 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:13:13.746 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2024-09-01T02:13:13.747 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5zdNmArSBGxAAZ6ZX95MUJ6nBbsAF3nycvw== --osd-uuid 0c834b82-b111-4177-9652-f9b811e9f1b3 2024-09-01T02:13:13.776 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:13.775+0000 7f805400f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:13.796 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:13.795+0000 7f805400f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:13.797 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:13.796+0000 7f805400f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:13.797 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:13.796+0000 7f805400f640 -1 bdev(0x56433b5efc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:13:13.797 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:13.796+0000 7f805400f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-01T02:13:19.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-01T02:13:19.467 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:13:19.468 INFO:tasks.workunit.client.0.smithi037.stdout:adding osd0 key to auth repository 2024-09-01T02:13:19.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:13:19.468 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:13:20.037 INFO:tasks.workunit.client.0.smithi037.stdout:start osd.0 2024-09-01T02:13:20.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:13:20.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=86276a7c-8c20-418a-a6ea-d066455d9b28 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.66036/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:13:20.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:13:20.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:13:20.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:13:20.064 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:20.063+0000 7f58db92c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:20.066 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:20.066+0000 7f58db92c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:20.067 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:20.067+0000 7f58db92c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:13:20.480 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:13:20.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:13:20.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:13:20.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:13:20.481 INFO:tasks.workunit.client.0.smithi037.stdout:0 2024-09-01T02:13:20.481 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:13:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:13:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:20.482 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:20.914 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:21.416 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:21.416+0000 7f58db92c640 -1 Falling back to public interface 2024-09-01T02:13:21.915 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:21.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:21.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:13:21.916 INFO:tasks.workunit.client.0.smithi037.stdout:1 2024-09-01T02:13:21.916 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:21.917 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:22.357 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:23.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:23.359 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:23.359 INFO:tasks.workunit.client.0.smithi037.stdout:2 2024-09-01T02:13:23.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:13:23.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:23.360 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:23.769 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:24.229 INFO:tasks.workunit.client.0.smithi037.stderr:2024-09-01T02:13:24.228+0000 7f58db92c640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:13:24.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:24.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:24.770 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:13:24.771 INFO:tasks.workunit.client.0.smithi037.stdout:3 2024-09-01T02:13:24.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:24.771 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:25.342 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:26.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:26.344 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:26.344 INFO:tasks.workunit.client.0.smithi037.stdout:4 2024-09-01T02:13:26.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:13:26.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:26.345 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:26.762 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:27.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:27.763 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:27.763 INFO:tasks.workunit.client.0.smithi037.stdout:5 2024-09-01T02:13:27.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:13:27.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:27.764 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:28.201 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:29.202 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:29.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:29.203 INFO:tasks.workunit.client.0.smithi037.stdout:6 2024-09-01T02:13:29.203 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:13:29.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:29.204 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:29.626 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:30.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:30.628 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:30.628 INFO:tasks.workunit.client.0.smithi037.stdout:7 2024-09-01T02:13:30.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:13:30.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:30.629 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:31.121 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:32.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:32.123 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:32.123 INFO:tasks.workunit.client.0.smithi037.stdout:8 2024-09-01T02:13:32.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:13:32.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:32.124 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:32.544 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:33.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:33.545 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:33.546 INFO:tasks.workunit.client.0.smithi037.stdout:9 2024-09-01T02:13:33.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:13:33.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:33.546 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:34.035 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:35.036 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:35.037 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:35.037 INFO:tasks.workunit.client.0.smithi037.stdout:10 2024-09-01T02:13:35.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:13:35.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:35.038 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:35.450 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:36.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:36.451 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:36.452 INFO:tasks.workunit.client.0.smithi037.stdout:11 2024-09-01T02:13:36.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:13:36.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:36.452 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:37.127 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:38.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:38.129 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:38.129 INFO:tasks.workunit.client.0.smithi037.stdout:12 2024-09-01T02:13:38.130 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:13:38.130 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:38.130 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:39.080 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:40.081 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:40.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:40.082 INFO:tasks.workunit.client.0.smithi037.stdout:13 2024-09-01T02:13:40.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:13:40.082 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:40.083 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:40.979 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:41.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:41.980 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:41.980 INFO:tasks.workunit.client.0.smithi037.stdout:14 2024-09-01T02:13:41.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:13:41.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:41.981 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:42.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:43.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:43.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:43.948 INFO:tasks.workunit.client.0.smithi037.stdout:15 2024-09-01T02:13:43.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:13:43.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:43.948 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:44.378 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:45.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:45.379 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:45.380 INFO:tasks.workunit.client.0.smithi037.stdout:16 2024-09-01T02:13:45.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:13:45.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:45.380 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:45.820 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:13:46.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:13:46.822 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:13:46.822 INFO:tasks.workunit.client.0.smithi037.stdout:17 2024-09-01T02:13:46.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:13:46.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:13:46.823 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:13:47.263 INFO:tasks.workunit.client.0.smithi037.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1923626444,v1:127.0.0.1:6803/1923626444] [v2:127.0.0.1:6804/1923626444,v1:127.0.0.1:6805/1923626444] exists,up 0c834b82-b111-4177-9652-f9b811e9f1b3 2024-09-01T02:13:47.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:13:47.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:13:47.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:13:47.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1860: test_repair: create_rbd_pool 2024-09-01T02:13:47.264 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-01T02:13:47.695 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' does not exist 2024-09-01T02:13:47.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-01T02:13:47.706 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-01T02:13:48.228 INFO:tasks.workunit.client.0.smithi037.stderr:pool 'rbd' already exists 2024-09-01T02:13:48.239 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:13:49.240 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-01T02:13:50.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1861: test_repair: wait_for_clean 2024-09-01T02:13:50.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:13:50.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:13:50.402 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:13:50.402 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:13:50.403 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:13:50.403 INFO:tasks.workunit.client.0.smithi037.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:13:50.403 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:13:50.403 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:13:50.403 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:13:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:13:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:13:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:13:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:13:50.498 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:13:50.499 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:13:50.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T02:13:50.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:13:50.954 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:13:50.954 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:13:51.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-01T02:13:51.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-01T02:13:51.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-01T02:13:51.070 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:13:51.071 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-01T02:13:51.071 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:13:51.071 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:13:51.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-01T02:13:51.072 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:13:51.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-01T02:13:51.073 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-01T02:13:51.073 INFO:tasks.workunit.client.0.smithi037.stdout:waiting osd.0 seq 21474836488 2024-09-01T02:13:51.073 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:13:51.505 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-01T02:13:51.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:13:52.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:13:52.507 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:13:52.955 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-01T02:13:52.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:13:52.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:13:52.956 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:13:53.538 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:13:53.933 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-01T02:13:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:13:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:13:53.934 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: repair 1.0 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:54.489 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:54.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:54.902 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-01T02:13:55.283 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:55.297 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:55.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:55.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:55.298 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:55.300 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:55.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:48.041851+0000 '>' 2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:55.689 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:56.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:56.690 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:56.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:56.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:56.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:56.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:56.691 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:57.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:48.041851+0000 '>' 2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:57.096 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:13:58.097 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:13:58.098 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-01T02:13:58.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:13:58.098 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:13:58.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:13:58.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:13:58.099 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:13:58.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:55.474562+0000 '>' 2024-09-01T02:13:48.041851+0000 2024-09-01T02:13:58.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-01T02:13:58.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-01T02:14:00.506 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-01T02:14:00.507 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:14:00.507 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:14:00.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:14:00.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:14:00.507 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:14:00.614 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: TIMEOUT=1 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: repair 1.0 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:14:00.615 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:14:01.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-01T02:13:55.474562+0000 2024-09-01T02:14:01.022 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-01T02:14:01.404 INFO:tasks.workunit.client.0.smithi037.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-01T02:14:01.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-01T02:13:55.474562+0000 2024-09-01T02:14:01.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-01T02:14:01.416 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T02:13:55.474562+0000 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-01T02:14:01.417 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-01T02:14:01.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T02:13:55.474562+0000 '>' 2024-09-01T02:13:55.474562+0000 2024-09-01T02:14:01.832 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-01T02:14:02.833 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-01T02:14:02.833 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 1 )) 2024-09-01T02:14:02.833 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: return 1 2024-09-01T02:14:02.833 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: return 1 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: teardown td/ceph-helpers 1 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:14:04.835 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:14:04.836 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:14:04.942 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:14:04.942 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:14:04.943 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:14:04.944 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:14:04.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:14:04.945 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:14:04.945 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:14:04.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:14:04.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:14:04.946 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:14:04.947 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:14:04.947 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:14:04.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-01T02:14:04.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-01T02:14:04.949 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-01T02:14:04.950 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-01T02:14:04.951 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2024-09-01T02:14:04.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:14:04.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:14:04.955 INFO:tasks.workunit.client.0.smithi037.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.66036 2024-09-01T02:14:04.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.66036 2024-09-01T02:14:04.956 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:14:04.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:14:04.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: return 1 2024-09-01T02:14:04.957 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: : exit 1 2024-09-01T02:14:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:14:04.958 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-01T02:14:04.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-01T02:14:05.265 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-09-01T02:14:05.448 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6c58fa51217e4128918dcf20d5c236d3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2024-09-01T02:14:05.450 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-01T02:14:05.459 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-01T02:14:05.460 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-01T02:14:05.514 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-01T02:14:05.523 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-01T02:14:05.524 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-01T02:14:05.542 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +601us[ +602us] +/- 57ms 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 -7277ns[-7277ns] +/- 57ms 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -867us[ -867us] +/- 80ms 2024-09-01T02:14:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T02:14:05.547 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-01T02:14:05.556 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-01T02:14:05.556 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-01T02:14:05.600 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-01T02:14:05.644 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-01T02:14:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:14:05.671 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-01T02:14:05.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-01T02:14:05.765 INFO:teuthology.task.internal:Duration was 2508.311217 seconds 2024-09-01T02:14:05.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-01T02:14:05.772 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-01T02:14:05.773 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-01T02:14:05.811 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-01T02:14:05.968 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-01T02:14:05.969 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-09-01T02:14:05.969 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-01T02:14:06.031 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-01T02:14:06.031 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-01T02:14:06.097 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-01T02:14:06.097 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-01T02:14:06.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-01T02:14:06.207 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-01T02:14:06.207 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-01T02:14:06.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-01T02:14:06.259 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:14:06.294 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-09-01T02:14:06.337 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:14:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:14:06.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-01T02:14:06.401 INFO:teuthology.task.internal:Transferring archived files... 2024-09-01T02:14:06.402 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884153/remote/smithi037 2024-09-01T02:14:06.403 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-01T02:14:06.762 INFO:teuthology.task.internal:Removing archive directory... 2024-09-01T02:14:06.762 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-01T02:14:06.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-01T02:14:06.813 INFO:teuthology.task.internal:Not uploading archives. 2024-09-01T02:14:06.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-01T02:14:06.822 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-01T02:14:06.822 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-01T02:14:06.860 INFO:teuthology.orchestra.run.smithi037.stdout: 262217 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 02:14 /home/ubuntu/cephtest 2024-09-01T02:14:06.860 INFO:teuthology.orchestra.run.smithi037.stdout: 394743 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 01:36 /home/ubuntu/cephtest/mnt.0 2024-09-01T02:14:06.860 INFO:teuthology.orchestra.run.smithi037.stdout: 394744 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 01:55 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-01T02:14:06.861 INFO:teuthology.orchestra.run.smithi037.stdout: 394787 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 01:55 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T02:14:06.861 INFO:teuthology.orchestra.run.smithi037.stdout: 409599 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 1 02:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-01T02:14:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:14:06.862 INFO:teuthology.orchestra.run.smithi037.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-01T02:14:06.862 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-01T02:14:06.867 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-01T02:14:06.877 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-01T02:14:06.886 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi037 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-01T02:14:06.890 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 2508.3112173080444 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi037 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6c58fa51217e4128918dcf20d5c236d3 status: fail success: false 2024-09-01T02:14:06.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-01T02:14:07.002 INFO:teuthology.run:FAIL